Okay, here’s the analysis of the provided text, following your specified format:
Keywords: PyTorch, Tensor, Autograd, Internals, C++
Overview: This blog post by Edward Z. Yang provides an overview of PyTorch internals, aimed at those who want to contribute to the project but are intimidated by the codebase. It explains the conceptual structure of a tensor library with automatic differentiation, covering topics like tensor data types, strides, storage, and the dispatch mechanisms for operations. The post also discusses tensor extensions (device, layout, dtype) and the autograd engine. Finally, it delves into the practical aspects of navigating the PyTorch codebase, including directory structure, code generation, kernel writing tools, and efficient development practices. The author encourages contributions to PyTorch and offers guidance on where to start.
Section Summaries:
- The tensor is the central data structure in PyTorch: Tensors consist of data and metadata like size, dtype, and device. Strides are a distinctive feature, defining the physical memory layout for views. Tensors and Storages are decoupled, allowing multiple tensors to share the same storage.
- Since we have been talking about Tensor, I also want to take a little time to the world of tensor extensions: PyTorch offers four extension points on tensors: device, layout, and dtype. These parameters uniquely determine what a tensor is. Wrapper classes can also extend Tensor functionality, especially when autograd is not required.
- I told you all about tensors, but if that was the only thing PyTorch provided, we’d basically just be a Numpy clone: Autograd is the machinery for automatic differentiation. It computes gradients by walking the forward computations backward. Variables wrap tensors and store autograd metadata.
- Enough about concepts, let’s look at some code: The key directories are torch/, torch/csrc/, aten/, and c10/. When you call a function like torch.add, there are multiple dispatches: Python to C++, variable dispatch, device type/layout dispatch, and finally the kernel.
- PyTorch offers a lot of useful tools for prospective kernel writers: A kernel consists of metadata, error checking, result tensor allocation, dtype dispatch, parallelization, and data access. Schemas define the operator’s type and bindings. Error checking can be done with TORCH_CHECK or higher-level APIs.
- To wrap up, I want to talk a little bit about working efficiently on PyTorch: Editing headers can cause long rebuilds. Use CI or set up a local development environment for testing. Ccache is highly recommended.
Related Tools:
- Stride Visualizer: https://ezyang.github.io/stride-visualizer/index.html
References:
- CONTRIBUTING: https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#codebase-structure
- Variable-Tensor merge in C++: https://github.com/pytorch/pytorch/issues/13638
- README in native: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md
- derivatives.yaml: https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml
- Dispatch.h: https://github.com/pytorch/pytorch/blob/21ef4cc615a7d9d772ade52a5023900718b09e92/aten/src/ATen/Dispatch.h#L62
- use-ccache: https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#use-ccache
- download and run the Docker images locally: https://github.com/pytorch/ossci-job-dsl
- high priority: https://github.com/pytorch/pytorch/issues?q=is%3Aopen+is%3Aissue+label%3A%22high+priority%22+label%3Atriaged
- autograd: https://github.com/pytorch/pytorch/issues?q=is%3Aopen+is%3Aissue+label%3Atriaged+label%3A%22module%3A+autograd%22
- small: https://github.com/pytorch/pytorch/issues?q=is%3Aopen+is%3Aissue+label%3Atriaged+label%3Asmall
Original Article Link: https://blog.ezyang.com/2019/05/pytorch-internals/
Consistency Check: The summary, keywords, section summaries, related tools, and references are all consistent with the original text and maintain the same order of information. There are no apparent contradictions or misrepresentations.