Neural Network Programming - Deep Learning with PyTorch

with deeplizard.

PyTorch Explained - Python Deep Learning Neural Network API

September 5, 2018 by

Blog

PyTorch - Python deep learning neural network API

Welcome back to this series on neural network programming with PyTorch. To kick this series off, let’s introduce PyTorch, a deep learning neural network package for Python. There’s no better place to start as we’ll be using PyTorch in this series to program our neural networks. Without further ado, let’s get started.

pytorch logo

PyTorch is a deep learning framework and a scientific computing package. This is how the PyTorch core team describes PyTorch, anyway. The scientific computing aspect of PyTorch is primarily a result PyTorch’s tensor library and associated tensor operations.

A tensor is an n-dimensional array.

For example, PyTorch torch.Tensor objects that are created from NumPy ndarray objects, share memory. This makes the transition between PyTorch and NumPy very cheap from a performance perspective.

With PyTorch tensors, GPU support is built-in. It’s very easy with PyTorch to move tensors to and from a GPU if we have one installed on our system.

PyTorch tensor operations can be performed on a GPU.

We’ll talk more about GPUs and why we use them in deep learning in the post on CUDA. For now, just know that PyTorch tensors and their associated operations are very similar to NumPy n-dimensional arrays.

Tensors are super important for deep learning and neural networks because they are the data structure that we ultimately use for building and training our neural networks.

On top of the tensor library, PyTorch has much more to offer in terms of building and training neural networks. Before we touch on the deep learning specifics of PyTorch, let’s look at some details on how PyTorch was created.

PyTorch: A brief history

The initial release of PyTorch was in October of 2016, and before PyTorch was created, there was and still is, another framework called Torch. Torch is a machine learning framework that’s been around for quite a while and is based on the Lua programming language.

The connection between PyTorch and this Lua version, called Torch, exists because many of the developers who maintain the Lua version are the individuals who created PyTorch.

python logo

Soumith Chintala is credited with bootstrapping the PyTorch project, and his reason for creating PyTorch is pretty simple, the Lua version of Torch was aging, and so a newer version written in Python was needed. As a result, PyTorch came to be.

Facebook created PyTorch

One thing that you may hear about PyTorch is that it was created and is maintained by Facebook. This is because Soumith Chintala worked at Facebook AI Research when PyTorch was created (still does at the time of this writing). However, there are many other companies with a vested interest in PyTorch.

facebook logo

The PyTorch GitHub repo indicates that there are quite a few contributors, upwards of seven hundred at the current moment. Right near the top of the contributions by commit, we have Soumith, but there are many others.

Let’s look now at the deep learning features of PyTorch.

Deep learning with PyTorch

This table gives us a list of PyTorch packages and their corresponding descriptions. These are the primary PyTorch components we’ll be learning about and using as we build neural networks in this series.

Package Description
torch The top-level PyTorch package and tensor library.
torch.nn A subpackage that contains modules and extensible classes for building neural networks.
torch.autograd A subpackage that supports all the differentiable Tensor operations in PyTorch.
torch.nn.functional A functional interface that contains typical operations used for building neural networks like loss functions, activation functions, and convolution operations.
torch.optim A subpackage that contains standard optimization operations like SGD and Adam.
torch.utils A subpackage that contains utility classes like data sets and data loaders that make data preprocessing easier.
torchvision A package that provides access to popular datasets, model architectures, and image transformations for computer vision.

At the moment, the torchvision package is separate from the top-level torch package. However, this may change in the future if torchvision is pulled in as a subpackage of torch.

Why use PyTorch for deep learning?

Let’s talk about the prospects for learning PyTorch. For beginners to deep learning and neural networks, the top reason for learning PyTorch is that it is a thin framework that stays out of the way.

PyTorch is thin and stays out of the way!

When we build neural networks with PyTorch, we are super close to programming neural networks from scratch. The experience of programming in PyTorch is as close as it gets to the real thing.

PyTorch is as close as it gets to the real thing!

After understanding the process of programming neural networks with PyTorch, it’s pretty easy to see how the process works from scratch in say pure Python. This is why PyTorch is great for beginners.

After using PyTorch, you’ll have a much deeper understanding of neural networks and the deep learning. One of the top philosophies of PyTorch is to stay out of the way, and this makes it so that we can focus on neural networks and less on the actual framework.

Philosophy of PyTorch

As of the writing of this post, PyTorch's development is guided by the following list:

  1. Stay out of the way
  2. Cater to the impatient
  3. Promote linear code-flow
  4. Full interop with the Python ecosystem
  5. Be as fast as anything else

The fact that PyTorch stays out of the way makes PyTorch well suited for deepening our understanding of neural networks. When we write PyTorch code, we are just writing and extending standard Python classes, and when we debug PyTorch code, we are using the standard Python debugger.

PyTorch’s design is modern, Pythonic, and thin. The source code is easy to read for Python developers because it’s written mostly in Python, and only drops into C++ and CUDA code for operations that are performance bottlenecks.

Overall, PyTorch is a great tool for deepening our understanding of deep learning and neural networks.

Investing in PyTorch as a deep learning framework

From a knowledge investment perspective, PyTorch can be seen as a safer option simply because Facebook is backing it, and it’s built for Python, which unlike Lua, has a large and growing deep learning community.

In addition to the Facebook and Python edge, PyTorch is super thin and highly integrated with Python and very thin, which makes it more likely that PyTorch will be capable of adapting to the rapidly evolving deep learning environment as things change over time.

These characteristics promote the longevity of PyTorch as a deep learning framework.

PyTorch for deep learning research

A common PyTorch characteristic that often pops up is that it’s great for research. The reason for this research suitability has do do with a technical design consideration. To optimize neural networks, we need to calculate derivatives, and to do this computationally, deep learning frameworks use what are called computational graphs.

Computational graphs are used to graph the function operations that occur on tensors inside neural networks.

These graphs are then used to compute the derivatives needed to optimize the neural network. PyTorch uses a computational graph that is called a dynamic computational graph. This means that the graph is generated on the fly as the operations are created.

This is in contrast to static graphs that are fully determined before the actual operations occur.

It just so happens that many of the cutting edge research topics in deep learning are requiring or benefiting greatly from dynamic graphs.

Convolutional neural network project in PyTorch

The first project that we will tackle in this series using PyTorch will be to build a convolutional neural network for classifying images from the Fashion-MNIST dataset.

fashion articles

This dataset contains a training set of sixty thousand examples from ten different classes of clothing items. We will use PyTorch to build a convolutional neural network that can accurately predict the correct article of clothing given an input piece, so stay tuned!

Let’s get ready to move forward with deep learning and neural networks. In the next post, we will get PyTorch installed. I’ll see you there!

Description

PyTorch is a deep learning framework and a scientific computing package. There’s no better place to start as we’ll be using PyTorch in this series to program our neural networks. Check out the corresponding blog and other resources for this video at: http://deeplizard.com/learn/video/iTKbyFh-7GM Code: https://www.patreon.com/posts/code-for-pytorch-21607032 ❤️🦎 Special thanks to the following polymaths of the deeplizard hivemind: Ruicong Xie Najib Akram Support collective intelligence, and join the deeplizard hivemind: http://deeplizard.com/hivemind Follow deeplizard: YouTube: https://www.youtube.com/deeplizard Twitter: https://twitter.com/deeplizard Facebook: https://www.facebook.com/Deeplizard-145413762948316 Steemit: https://steemit.com/@deeplizard Instagram: https://www.instagram.com/deeplizard/ Pinterest: https://www.pinterest.com/deeplizard/ Check out products deeplizard suggests on Amazon: https://www.amazon.com/shop/deeplizard Support deeplizard with crypto: Bitcoin: 1AFgm3fLTiG5pNPgnfkKdsktgxLCMYpxCN Litecoin: LTZ2AUGpDmFm85y89PFFvVR5QmfX6Rfzg3 Ether: 0x9105cd0ecbc921ad19f6d5f9dd249735da8269ef Recommended books on AI: The Most Human Human: What Artificial Intelligence Teaches Us About Being Alive: http://amzn.to/2GtjKqu Life 3.0: Being Human in the Age of Artificial Intelligence https://amzn.to/2H5Iau4 #ai #pytorch #deeplearning Follow Soumith: https://twitter.com/soumithchintala Playlists: Data Science - https://www.youtube.com/playlist?list=PLZbbT5o_s2xrth-Cqs_R9-us6IWk9x27z Machine Learning - https://www.youtube.com/playlist?list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras - https://www.youtube.com/playlist?list=PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL TensorFlow.js - https://www.youtube.com/playlist?list=PLZbbT5o_s2xr83l8w44N_g3pygvajLrJ- PyTorch - https://www.youtube.com/watch?v=v5cngxo4mIg&list=PLZbbT5o_s2xrfNyHZsM6ufI0iZENK9xgG Music: Thinking Music by Kevin MacLeod Jarvic 8 by Kevin MacLeod YouTube: https://www.youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ Website: http://incompetech.com/ Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/