Machine Learning & Deep Learning Fundamentals

with deeplizard.

Batch Size in a Neural Network explained

November 22, 2017 by

Blog

Batch size in artificial neural networks

In this post, we’ll discuss what it means to specify a batch size as it pertains to training an artificial neural network, and we’ll also see how to specify the batch size for our model in code using Keras.

gears

In our previous post on how an artificial neural network learns, we saw that when we train our model, we have to specify a batch size. Let’s go ahead and discuss the details about this now.

Introducing batch size

Put simply, the batch size is the number of samples that will be passed through to the network at one time. Note that a batch is also commonly referred to as a mini-batch.

The batch size is the number of samples that are passed to the network at once.

Now, recall that an epoch is one single pass over the entire training set to the network. The batch size and an epoch are not the same thing. Let’s illustrate this with an example.

Batches in an epoch

Let’s say we have 1000 images of dogs that we want to train our network on in order to identify different breeds of dogs. Now, let’s say we specify our batch size to be 10. This means that 10 images of dogs will be passed as a group, or as a batch, at one time to the network.

Given that a single epoch is one single pass of all the data through the network, it will take 100 batches to make up full epoch. We have 1000 images divided by a batch size of 10, which equals 100 total batches.

batches in epoch = training set size / batch_size

Ok, we have the idea of batch size down now, but what’s the point? Why not just pass each data element one-by-one to our model rather than grouping the data in batches?

Why use batches?

Well, for one, generally the larger the batch size, the quicker our model will complete each epoch during training. This is because, depending on our computational resources, our machine may be able to process much more than one single sample at a time.

The trade-off, however, is that even if our machine can handle very large batches, the quality of the model may degrade as we set our batch larger and may ultimately cause the model to be unable to generalize well on data it hasn't seen before.

In general, the batch size is another one of the hyperparameters that we must test and tune based on how our specific model is performing during training. This parameter will also have to be tested in regards to how our machine is performing in terms of its resource utilization when using different batch sizes.

server

For example, if we were to set our batch size to a relatively high number, say 100, then our machine may not have enough computational power to process all 100 images in parallel, and this would suggest that we need to lower our batch size.

Alright, we should now have a general idea about what batch size is. Let’s see how we specify this parameter in code now using Keras.

keras logo

Working with batch size in Keras

We’ll be working with the same model we’ve used in the last several posts. This is just an arbitrary Sequential model.

model = Sequential([
    Dense(16, input_shape=(1,), activation='relu'),
    Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)),
    Dense(2, activation='sigmoid')
])

Let’s focus our attention on where we call model.fit(). We know this is the function we call to train our model, and we saw this in action in our previous post on how an artificial neural network learns.

model.fit(
    scaled_train_samples, 
    train_labels, 
    validation_data=valid_set, 
    batch_size=10,
    epochs=20, 
    shuffle=True, 
    verbose=2
)

This fit() function accepts a parameter called batch_size. This is where we specify our batch_size for training. In this example, we’ve just arbitrarily set the value to 10.

Now, during the training of this model, we’ll be passing in 10 samples at a time until we eventually pass in all the training data to complete one single epoch. Then, we’ll start the same process over again to complete the next epoch.

That’s really all there is to it for specifying the batch size for training a model in Keras!

Wrapping up

Hopefully now we have a general understanding of what the batch size is and how to specify it in Keras. I’ll see you in the next one!

Description

In this video, we explain the concept of the batch size used during training of an artificial neural network and also show how to specify the batch size in code with Keras. Blog: http://deeplizard.com/learn/video/U4WB9p6ODjM Follow deeplizard on Twitter: https://twitter.com/deeplizard Follow deeplizard on Steemit: https://steemit.com/@deeplizard Become a patron: https://www.patreon.com/deeplizard Support deeplizard: Bitcoin: 1AFgm3fLTiG5pNPgnfkKdsktgxLCMYpxCN Litecoin: LTZ2AUGpDmFm85y89PFFvVR5QmfX6Rfzg3 Ether: 0x9105cd0ecbc921ad19f6d5f9dd249735da8269ef Recommended books: The Most Human Human: What Artificial Intelligence Teaches Us About Being Alive: http://amzn.to/2GtjKqu