Deep Learning Deployment Fundamentals

Initializing and accessing bias in Keras

In this episode, we'll see how we can initialize and access the biases in a neural network in code with Keras. Let's get to it!

Keras model

Let's have a look at this arbitrary small neural network with one hidden Dense layer containing 4 nodes, and an output layer with 2 nodes.

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential([
    Dense(4, input_shape=(1,), activation='relu', use_bias=True, bias_initializer='zeros'),
    Dense(2, activation='softmax')
])

Everything here is pretty standard, and you should be familiar with almost all of it based on previous episodes in this series. The only new items are in in the hidden layer where we have two parameters we haven't yet seen before: use_bias and bias_initializer.

In another episode, we've discussed what exactly bias is in a neural network. Now, we're going to specifically focus on how bias is worked with within a Keras model.

Parameter: use_bias

In Keras, we specify whether or not we want a given layer to include biases for all of its neurons with the use_bias parameter. If we do want to include biases, we set the parameter value to True. Otherwise, we set it to False.

The default value is True, so if we don't specify this parameter here at all, the layer will include bias terms by default.

Parameter: bias_initializer

Next, we have the bias_initializer parameter, which determines how the biases are initialized. This initialization process is really similar to the weight initialization process that we talked about in another episode.

This parameter determines how the biases are first set before we start training the model.

We are setting this parameter's value to the string 'zeros'. This means that all 4 biases in this layer will be set to a value of 0 before the model starts training.

'zeros' is actually the default value for the bias_initializer parameter. If we instead wanted to change this so that the biases were set to some other type of values, like all ones, or random numbers, then we can.

Keras has a list of initializers that it supports and they're actually the same list of initializers we talked about when we discussed weight initialization. So we could even initialize the biases with Xavier initialization if we wanted.

Observing initialized bias terms

After we initialize these biases, we can check them out and inspect at their values by calling model.get_weights.

model.get_weights()

[array([[-0.733, 0.017, 0.561, 0.331]], dtype-float32),
    array([0.,0.,0.,0.], dtype=float32),
    array([[-0.281, 0.502],
        [0.081, 0.713],
        [0.004, -0.696],
        [0.013, -0.488]], dtype=float32),
    array([0., 0.], dtype=float32)]

This gives us all the weights and all the biases for each layer in the model. We can see that we have these randomly initialized weights in the weight matrix for the first hidden layer, and we also have the bias vector containing 4 zeros corresponding to the bias term for each node in the layer for which we specified 'zeros' as the bias_initializer.

Similarly, we have the weight matrix corresponding to the output layer, which is again, followed by the bias vector that contains 2 zeros corresponding to the bias term for each node in this layer.

Remember, we didn't set any bias parameters for the output layer, but because Keras uses bias and initializes bias terms with zeros by default, we get this for free.

After being initialized, during training, these biases (and weights) will be updated as the model learns the optimized values for them. If we were to train this model and then call the get_weights() function again, then the values for the weights and biases would likely be very different.

Wrapping up

As we now know, our Keras models have been using bias this whole time without any effort from our side since, by default, Keras is initializing the biases with zeros. We showed this for Dense layers but the same is true for other layer types as well, like convolutional layers for example.

After first learning about bias on a fundamental level and now seeing in applied in code, what are your thoughts? See ya in the next one!