Neural Network Programming - Deep Learning with PyTorch

with deeplizard.

Rank, Axes, and Shape Explained - Tensors for Deep Learning

September 13, 2018 by

Blog

Rank, Axes and Shape - Tensors for deep learning

Welcome back to this series on neural network programming with PyTorch. In this post, we will dig in deeper with tensors and introduce three fundamental tensor attributes, rank, axes, and shape. Without further ado, let's get started.

cyborg

The concepts of rank, axes, and shape are the tensor attributes that will concern us most in deep learning.

  • Rank
  • Axes
  • Shape

The rank, axes, and shape are three tensor attributes that will concern us most when starting out with tensors in deep learning. These concepts build on one another starting with rank, then axes, and building up to shape, so keep any eye out for this relationship between these three.

tensor cube

Together rank, axes, and shape are all fundamentally connected to the concept of indexes that we discussed in the previous post. If you haven’t seen that one, I highly recommend you check it out. Let’s kick things off starting at the ground floor by introducing the rank of a tensor.

Rank of a tensor

The rank of a tensor refers to the number of dimensions present within the tensor. Suppose we are told that we have a rank-2 tensor. This means all of the following:

  • We have a matrix
  • We have a 2d-array
  • We have a 2d-tensor

We are introducing the word rank here because it is commonly used in deep learning when referring to the number of dimensions present within a given tensor. This is just another one of the instances where different areas of study use different words to refer to the same concept. Don't let it throw you off!

Rank and indexes

The rank of a tensor tells us how many indexes are required to access (refer to) a specific data element contained within the tensor data structure.

A tensor's rank tells us how many indexes are needed to refer to a specific element within the tensor.

Let’s build on the concept of rank by looking at the axes of a tensor.

Axes of a tensor

If we have a tensor, and we want to refer to a specific dimension, we use the word axis in deep learning.

An axis of a tensor is a specific dimension of a tensor.

If we say that a tensor is a rank 2 tensor, we mean that the tensor has 2 dimensions, or equivalently, the tensor has two axes.

Elements are said to exist or run along an axis. This running is constrained by the length of each axis. Let's look at the length of an axis now.

Length of an axis

The length of each axis tells us how many indexes are available along each axis.

Suppose we have a tensor called t, and we know that the first axis has a length of three while the second axis has a length of four.

Since the first axis has a length of three, this means that we can index three positions along the first axis like so:

t[0]
t[1]
t[2]

All of these indexes are valid, but we can't move passed index 2.

Since the second axis has a length of four, we can index four positions along the second axis. This is possible for each index of the first axis, so we have

t[0][0]
t[1][0]
t[2][0]

t[0][1]
t[1][1]
t[2][1]

t[0][2]
t[1][2]
t[2][2]

t[0][3]
t[1][3]
t[2][3]

Tensor axes example

Let's look at some examples to make this solid. We'll consider the same tensor dd as before:

> dd = [
[1,2,3],
[4,5,6],
[7,8,9]
]

Each element along the first axis, is an array:

> dd[0]
[1, 2, 3]

> dd[1]
[4, 5, 6]

> dd[2]
[7, 8, 9]

Each element along the second axis, is a number:

> dd[0][0]
1

> dd[1][0]
4

> dd[2][0]
7

> dd[0][1]
2

> dd[1][1]
5

> dd[2][1]
8

> dd[0][2]
3

> dd[1][2]
6

> dd[2][2]
9

Note that, with tensors, the elements of the last axis are always numbers. Every other axis will contain n-dimensional arrays. This is what we see in this example, but this idea generalizes.

The rank of a tensor tells us how many axes a tensor has, and the length of these axes leads us to the very important concept known as the shape of a tensor.

Shape of a tensor

The shape of a tensor is determined by the length of each axis, so if we know the shape of a given tensor, then we know the length of each axis, and this tells us how many indexes are available along each axis.

The shape of a tensor gives us the length of each axis of the tensor.

Let’s consider the same tensor dd as before:

> dd = [
[1,2,3],
[4,5,6],
[7,8,9]
]

To work with this tensor's shape, we’ll create a torch.Tensor object like so:

> t = torch.tensor(dd)
> t
tensor([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
])

> type(t)
torch.Tensor

Now, we have a torch.Tensor object, and so we can ask to see the tensor's shape:

> t.shape
torch.Size([3,3])

This allows us to see the tensor's shape is 3 x 3. Note that, in PyTorch, size and shape of a tensor are the same thing.

The shape of 3 x 3 tells us that each axis of this rank two tensor has a length of 3 which means that we have three indexes available along each axis. Let's look now at why the shape of a tensor is so important.

A tensor's shape is important

The shape of a tensor is important for a few reasons. The first reason is because the shape allows us to conceptually think about, or even visualize, a tensor. Higher rank tensors become more abstract, and the shape gives us something concrete to think about.

The shape also encodes all of the relevant information about axes, rank, and therefore indexes.

tensor operation

Additionally, one of the types of operations we must perform frequently when we are programming our neural networks is called reshaping.

As our tensors flow through our networks, certain shapes are expected at different points inside the network, and as neural network programmers, it is our job to understand the incoming shape and have the ability to reshape as needed.

Reshaping a tensor

Before we look at reshaping tensors, recall how we reshaped the list of terms we started with earlier:

Shape 6 x 1

  • number
  • scalar
  • array
  • vector
  • 2d-array
  • matrix

Shape 2 x 3

  • number, array, 2d-array
  • scalar, vector, matrix

Shape 3 x 2

  • number, scalar
  • array, vector
  • 2d-array, matrix

Each of these groups of terms represent the same underlying data only with differing shapes. This is just a little example to motivate the idea of reshaping.

The important take-away from this motivation is that the shape changes the grouping of the terms but does not change the underlying terms themselves.

Let’s look at our example tensor dd again:

> t = torch.tensor(dd)
> t
tensor([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
])

This torch.Tensor is a rank 2 tensor with a shape of [3,3] or 3 x 3.

Now, suppose we need to reshape t to be of shape [1,9]. This would give us one array along the first axis and nine numbers along the second axis:

> t.reshape(1,9)
tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9]])

> t.reshape(1,9).shape
torch.Size([1, 9])

Now, one thing to notice about reshaping is that the product of the component values in the shape must equal the total number of elements in the tensor.

For example:

  • 3 * 3 = 9
  • 1 * 9 = 9

This makes it so that there are enough positions inside the tensor data structure to contain all of the original data elements after the reshaping.

Reshaping changes the shape but not the underlying data elements.

This was just a light introduction to tensor reshaping. In a future post, we'll cover the concept in more detail.

Wrapping up

This gives a an introduction to tensors. We should now have a good understanding of tensors and the lingo used to describe them like rank, axes, and shape. Soon, we'll see the various ways we can create tensors in PyTorch. I’ll see you in the next one!

Description

Part 2: Introducing tensors for deep learning and neural network programming. fast.ai: http://www.fast.ai/ 💥🦎 DEEPLIZARD COMMUNITY RESOURCES 🦎💥 👀 OUR VLOG: 🔗 https://www.youtube.com/channel/UC9cBIteC3u7Ee6bzeOcl_Og 👉 Check out the blog post and other resources for this video: 🔗 https://deeplizard.com/learn/video/AiyK0idr4uM 💻 DOWNLOAD ACCESS TO CODE FILES 🤖 Available for members of the deeplizard hivemind: 🔗 https://www.patreon.com/posts/27743395 🧠 Support collective intelligence, join the deeplizard hivemind: 🔗 https://deeplizard.com/hivemind 🤜 Support collective intelligence, create a quiz question for this video: 🔗 https://deeplizard.com/create-quiz-question 🚀 Boost collective intelligence by sharing this video on social media! ❤️🦎 Special thanks to the following polymaths of the deeplizard hivemind: yasser Prash 👀 Follow deeplizard: Our vlog: https://www.youtube.com/channel/UC9cBIteC3u7Ee6bzeOcl_Og Twitter: https://twitter.com/deeplizard Facebook: https://www.facebook.com/Deeplizard-145413762948316 Patreon: https://www.patreon.com/deeplizard YouTube: https://www.youtube.com/deeplizard Instagram: https://www.instagram.com/deeplizard/ 🎓 Deep Learning with deeplizard: Fundamental Concepts - https://deeplizard.com/learn/video/gZmobeGL0Yg Beginner Code - https://deeplizard.com/learn/video/RznKVRTFkBY Advanced Code - https://deeplizard.com/learn/video/v5cngxo4mIg Advanced Deep RL - https://deeplizard.com/learn/video/nyjbcRQ-uQ8 🎓 Other Courses: Data Science - https://deeplizard.com/learn/video/d11chG7Z-xk Trading - https://deeplizard.com/learn/video/ZpfCK_uHL9Y 🛒 Check out products deeplizard recommends on Amazon: 🔗 https://www.amazon.com/shop/deeplizard 📕 Get a FREE 30-day Audible trial and 2 FREE audio books using deeplizard’s link: 🔗 https://amzn.to/2yoqWRn 🎵 deeplizard uses music by Kevin MacLeod 🔗 https://www.youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ 🔗 http://incompetech.com/ ❤️ Please use the knowledge gained from deeplizard content for good, not evil.