# Neural Network Programming - Deep Learning with PyTorch

with deeplizard.

## Rank, Axes, and Shape Explained - Tensors for Deep Learning

September 13, 2018 by

Blog

### Rank, Axes and Shape - Tensors for deep learning

Welcome back to this series on neural network programming with PyTorch. In this post, we will dig in deeper with tensors and introduce three fundamental tensor attributes, rank, axes, and shape. Without further ado, let's get started.

The concepts of rank, axes, and shape are the tensor attributes that will concern us most in deep learning.

• Rank
• Axes
• Shape

The rank, axes, and shape are three tensor attributes that will concern us most when starting out with tensors in deep learning. These concepts build on one another starting with rank, then axes, and building up to shape, so keep any eye out for this relationship between these three.

Together rank, axes, and shape are all fundamentally connected to the concept of indexes that we discussed in the previous post. If you haven’t seen that one, I highly recommend you check it out. Let’s kick things off starting at the ground floor by introducing the rank of a tensor.

### Rank of a tensor

The rank of a tensor refers to the number of dimensions present within the tensor. Suppose we are told that we have a rank-2 tensor. This means all of the following:

• We have a matrix
• We have a 2d-array
• We have a 2d-tensor

We are introducing the word rank here because it is commonly used in deep learning when referring to the number of dimensions present within a given tensor. This is just another one of the instances where different areas of study use different words to refer to the same concept. Don't let it throw you off!

#### Rank and indexes

The rank of a tensor tells us how many indexes are required to access (refer to) a specific data element contained within the tensor data structure.

A tensor's rank tells us how many indexes are needed to refer to a specific element within the tensor.

Let’s build on the concept of rank by looking at the axes of a tensor.

### Axes of a tensor

If we have a tensor, and we want to refer to a specific dimension, we use the word axis in deep learning.

An axis of a tensor is a specific dimension of a tensor.

If we say that a tensor is a rank `2` tensor, we mean that the tensor has `2` dimensions, or equivalently, the tensor has two axes.

Elements are said to exist or run along an axis. This running is constrained by the length of each axis. Let's look at the length of an axis now.

#### Length of an axis

The length of each axis tells us how many indexes are available along each axis.

Suppose we have a tensor called `t`, and we know that the first axis has a length of three while the second axis has a length of four.

Since the first axis has a length of three, this means that we can index three positions along the first axis like so:

```t[0]
t[1]
t[2]
```

All of these indexes are valid, but we can't move passed index `2`.

Since the second axis has a length of four, we can index four positions along the second axis. This is possible for each index of the first axis, so we have

```t[0][0]
t[1][0]
t[2][0]

t[0][1]
t[1][1]
t[2][1]

t[0][2]
t[1][2]
t[2][2]

t[0][3]
t[1][3]
t[2][3]
```

#### Tensor axes example

Let's look at some examples to make this solid. We'll consider the same tensor `dd` as before:

```> dd = [
[1,2,3],
[4,5,6],
[7,8,9]
]
```

Each element along the first axis, is an array:

```> dd[0]
[1, 2, 3]

> dd[1]
[4, 5, 6]

> dd[2]
[7, 8, 9]
```

Each element along the second axis, is a number:

```> dd[0][0]
1

> dd[1][0]
4

> dd[2][0]
7

> dd[0][1]
2

> dd[1][1]
5

> dd[2][1]
8

> dd[0][2]
3

> dd[1][2]
6

> dd[2][2]
9
```

Note that, with tensors, the elements of the last axis are always numbers. Every other axis will contain n-dimensional arrays. This is what we see in this example, but this idea generalizes.

The rank of a tensor tells us how many axes a tensor has, and the length of these axes leads us to the very important concept known as the shape of a tensor.

### Shape of a tensor

The shape of a tensor is determined by the length of each axis, so if we know the shape of a given tensor, then we know the length of each axis, and this tells us how many indexes are available along each axis.

The shape of a tensor gives us the length of each axis of the tensor.

Let’s consider the same tensor `dd` as before:

```> dd = [
[1,2,3],
[4,5,6],
[7,8,9]
]
```

To work with this tensor's shape, we’ll create a `torch.Tensor` object like so:

```> t = torch.tensor(dd)
> t
tensor([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
])

> type(t)
torch.Tensor
```

Now, we have a `torch.Tensor` object, and so we can ask to see the tensor's `shape`:

```> t.shape
torch.Size([3,3])
```

This allows us to see the tensor's shape is `3 x 3`. Note that, in PyTorch, size and shape of a tensor are the same thing.

The shape of `3 x 3` tells us that each axis of this rank two tensor has a length of `3` which means that we have three indexes available along each axis. Let's look now at why the shape of a tensor is so important.

#### A tensor's shape is important

The shape of a tensor is important for a few reasons. The first reason is because the shape allows us to conceptually think about, or even visualize, a tensor. Higher rank tensors become more abstract, and the shape gives us something concrete to think about.

The shape also encodes all of the relevant information about axes, rank, and therefore indexes.

Additionally, one of the types of operations we must perform frequently when we are programming our neural networks is called reshaping.

As our tensors flow through our networks, certain shapes are expected at different points inside the network, and as neural network programmers, it is our job to understand the incoming shape and have the ability to reshape as needed.

#### Reshaping a tensor

Before we look at reshaping tensors, recall how we reshaped the list of terms we started with earlier:

Shape `6 x 1`

• number
• scalar
• array
• vector
• 2d-array
• matrix

Shape `2 x 3`

• number, array, 2d-array
• scalar, vector, matrix

Shape `3 x 2`

• number, scalar
• array, vector
• 2d-array, matrix

Each of these groups of terms represent the same underlying data only with differing shapes. This is just a little example to motivate the idea of reshaping.

The important take-away from this motivation is that the shape changes the grouping of the terms but does not change the underlying terms themselves.

Let’s look at our example tensor dd again:

```> t = torch.tensor(dd)
> t
tensor([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
])
```

This `torch.Tensor` is a rank `2` tensor with a shape of `[3,3]` or `3 x 3`.

Now, suppose we need to reshape `t` to be of shape `[1,9]`. This would give us one array along the first axis and nine numbers along the second axis:

```> t.reshape(1,9)
tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9]])

> t.reshape(1,9).shape
torch.Size([1, 9])
```

Now, one thing to notice about reshaping is that the product of the component values in the shape must equal the total number of elements in the tensor.

For example:

• 3 * 3 = 9
• 1 * 9 = 9

This makes it so that there are enough positions inside the tensor data structure to contain all of the original data elements after the reshaping.

Reshaping changes the shape but not the underlying data elements.

This was just a light introduction to tensor reshaping. In a future post, we'll cover the concept in more detail.

### Wrapping up

This gives a an introduction to tensors. We should now have a good understanding of tensors and the lingo used to describe them like rank, axes, and shape. Soon, we'll see the various ways we can create tensors in PyTorch. I’ll see you in the next one!

Description

Part 2: Introducing tensors for deep learning and neural network programming. Check out the corresponding blog and other resources for this video at: http://deeplizard.com/learn/video/AiyK0idr4uM Code: https://www.patreon.com/posts/code-for-pytorch-21607032 ❤️🦎 Special thanks to the following polymaths of the deeplizard hivemind: Ruicong Xie Najib Akram Support collective intelligence, and join the deeplizard hivemind: http://deeplizard.com/hivemind Follow deeplizard: YouTube: https://www.youtube.com/deeplizard Twitter: https://twitter.com/deeplizard Facebook: https://www.facebook.com/Deeplizard-145413762948316 Steemit: https://steemit.com/@deeplizard Instagram: https://www.instagram.com/deeplizard/ Pinterest: https://www.pinterest.com/deeplizard/ Check out products deeplizard suggests on Amazon: https://www.amazon.com/shop/deeplizard Support deeplizard with crypto: Bitcoin: 1AFgm3fLTiG5pNPgnfkKdsktgxLCMYpxCN Litecoin: LTZ2AUGpDmFm85y89PFFvVR5QmfX6Rfzg3 Ether: 0x9105cd0ecbc921ad19f6d5f9dd249735da8269ef Recommended books on AI: The Most Human Human: What Artificial Intelligence Teaches Us About Being Alive: http://amzn.to/2GtjKqu Life 3.0: Being Human in the Age of Artificial Intelligence https://amzn.to/2H5Iau4 fast.ai: http://www.fast.ai/ Playlists: Data Science - https://www.youtube.com/playlist?list=PLZbbT5o_s2xrth-Cqs_R9-us6IWk9x27z Machine Learning - https://www.youtube.com/playlist?list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras - https://www.youtube.com/playlist?list=PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL TensorFlow.js - https://www.youtube.com/playlist?list=PLZbbT5o_s2xr83l8w44N_g3pygvajLrJ- PyTorch - https://www.youtube.com/watch?v=v5cngxo4mIg&list=PLZbbT5o_s2xrfNyHZsM6ufI0iZENK9xgG Music: Thinking Music by Kevin MacLeod Jarvic 8 by Kevin MacLeod Investigations by Kevin MacLeod YouTube: https://www.youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ Website: http://incompetech.com/ Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/