Deep Learning Deployment Fundamentals

Mapping Keras labels to image classes

There are many questions on Stackoverflow, Reddit, and the like all asking a particular question regarding image labeling in Keras. I've not addressed it before, so I wanted to just quickly touch on it in this episode.

This mysterious lingering question is...

Admittedly, this is a bit of a loaded question, especially without any context around it, but thankfully we'll see that the answer is pretty straightforward.

Before we get to the answer, though, let's first jog our memory about some previous topics we covered so we can gain some context and see where this question would come into play.

Understanding how Keras labels data

In a previous episode, we went through the set up to prepare a set of cat and dog image data for training a CNN.

train_batches = ImageDataGenerator().flow_from_directory
    (train_path, target_size=(224,224), classes=['dog', 'cat'], batch_size=10)
valid_batches = ImageDataGenerator().flow_from_directory
    (valid_path, target_size=(224,224), classes=['dog', 'cat'], batch_size=4)
test_batches = ImageDataGenerator().flow_from_directory
    (test_path, target_size=(224,224), classes=['dog', 'cat'], batch_size=10)

We created ImageDataGenerators for our training, validation, and test sets and called flow_from_directory() on each of these. We pointed to the location on disk where the corresponding data resides.

Recall that flow_from_directory() just creates batches of the image data to be used for training, validating, and testing.

We then utilized this plots() function, which allowed us to plot the images with the respective labels in our Jupyter notebook.

# plots images with labels within jupyter notebook
def plots(ims, figsize=(12,6), rows=1, interp=False, titles=None):
    if type(ims[0]) is np.ndarray:
        ims = np.array(ims).astype(np.uint8)
        if (ims.shape[-1] != 3):
            ims = ims.transpose((0,2,3,1))
    f = plt.figure(figsize=figsize)
    cols = len(ims)//rows if len(ims) % 2 == 0 else len(ims)//rows + 1
    for i in range(len(ims)):
        sp = f.add_subplot(rows, cols, i+1)
        sp.axis('Off')
        if titles is not None:
            sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i], interpolation=None if interp else 'none')

Using this function, we can see that since our data is categorical in nature, Keras has assigned the cat and dog classes these one-hot encoded vectors as its labels.

imgs, labels = next(train_batches)
plots(imgs, titles=labels)

For example, a cat is not referred to as “cat” but instead as [0,1]. Dog is referred to as [1,0].

Since we're only dealing with two classes here, we could have very well chosen to use binary classification, rather than categorical. If we used binary, then rather than the 2D labels that we have here, we would instead have 1D labels consisting of a 0 or a 1.

We know from visually inspecting the output above which labels Keras has assigned to each of the categorical classes of cat and dog. What if we didn't plot this, though? Then how will we know whether the label [0,1] corresponds to cat or dog?

This then begs the question why do we even care?

Recall that we previously called model.predict_generator() and passed in the test_batches and printed out the predictions.

predictions = model.predict_generator(test_batches, steps=1, verbose=0)
predictions

array([[ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.],
       [ 0.,  1.]], dtype=float32)

In this case, all of our predictions were [0,1]. Note, this was prior to tuning our model to perform better, so here it was performing poorly and classifying everything as a cat.

If we didn't print our batches ahead of time, though, we wouldn't know that a [0,1] vector mapped to a cat, and these predictions wouldn't make sense to us. We wouldn't know if the model was predicting cats or dogs.

We need to be able to understand which label belongs to which class without having to plot the data. It is this idea that is being brought up and asked about on forums. How do we know which labels belong to which classes?

Luckily, we have a very straight forward way to do this.

There is an attribute called class_indices that we can access on an ImageDataGenerator, which will return the dictionary that contains the mapping from class names to class indices.

Let's see what this looks like. We can call this on any of the ImageDatagenerators that we created, either test_batches, train_batches, or valid_batches.

We'll demo this with test_batches.

Running the code test_batches.class_indices gives us the following output.

{'cat': 1, 'dog': 0}

We see that the value 1 is assigned to cat, and the value 0 is assigned to dog.

But... our values are supposed to be one-hot encoded vectors, not just 0s and 1s. So, what's going on?

This output is actually giving us the index for where the corresponding value of a 1 resides in the one-hot encoded vectors.

So, it's showing that a cat corresponds to a 1. By looking at an image of a cat in the plot we discussed above, we can see the value of the 1 in the one-hot encoded label is indeed the first index of that vector.

For a dog, the value of 1 is in the zeroth index, which is why the class index for a dog is 0.

If we had another category, lizard, then we would have another element showing that the class lizard corresponds to the index of 2. Then, all of our vectors would be length 3 for having three categorical classes.

{'lizard': 2, 'cat': 1, 'dog': 0}

In this case, the dog label would be [1,0,0]. The cat label would be [0,1,0]. The lizard label would be [0,0,1].

Hopefully this clears up the confusion for how to access the labels that are assigned to the classes by Keras and why we need this information anyway.