PyTorch - Python Deep Learning Neural Network API

Max Pooling vs No Max Pooling

Welcome to deeplizard. My name is Chris. In this lesson, we're going to see how a neural network performs with and without max pooling.

Without further ado, let's get started.

Testing with and without Max Pooling

So far in this course, we've built a convolution neural network with max pooling, and we've been training on the Fashion MNIST dataset.

Since the testing framework we've built makes it easy to test various networks, we can easily train with and without max pooling operations and see how the results compare.

Let's start by creating the two network variants inside our NetworkFactory class.

class NetworkFactory():
    @staticmethod
    def get_network(name):
        if name == 'max_pool':
            torch.manual_seed(50)
            return nn.Sequential(
                    nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
                , nn.ReLU()
                , nn.MaxPool2d(kernel_size=2, stride=2)
                , nn.BatchNorm2d(6)
                , nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
                , nn.ReLU()
                , nn.MaxPool2d(kernel_size=2, stride=2)
                , nn.Flatten(start_dim=1)  
                , nn.Linear(in_features=12*4*4, out_features=120)
                , nn.ReLU()
                , nn.BatchNorm1d(120)
                , nn.Linear(in_features=120, out_features=60)
                , nn.ReLU()
                , nn.Linear(in_features=60, out_features=10)
            )
        elif name == 'no_max_pool':
            torch.manual_seed(50)
            return nn.Sequential(
                    nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
                , nn.ReLU()
                # , nn.MaxPool2d(kernel_size=2, stride=2)
                , nn.BatchNorm2d(6)
                , nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
                , nn.ReLU()
                # , nn.MaxPool2d(kernel_size=2, stride=2)
                , nn.Flatten(start_dim=1)  
                , nn.Linear(in_features=12*20*20, out_features=120)
                , nn.ReLU()
                , nn.BatchNorm1d(120)
                , nn.Linear(in_features=120, out_features=60)
                , nn.ReLU()
                , nn.Linear(in_features=60, out_features=10)
            )
        else:
            return None

Note that everything is the same in the 'no_max_pool' network except for the number of input features coming into the first linear layer. This is due to removal of the max pool operations that reduce the total number of features flowing through the network.

We now have 12*20*20 opposed to 12*4*4, so we can see that the 'no_max_pool' network has more output features coming from the convolutional part of the network.

We're ready now to test both of these networks with the following run configurations.

params = OrderedDict(
    lr = [.01]
    , batch_size = [1000]
    , num_workers = [1]
    , device = ['cuda']
    , network = ['max_pool', 'no_max_pool']
)

Noe that we are simply specifying the name of the two networks in the network section of the run configurations. These names will be passed to the NetworkFactory inside the run loop.

m = RunManager()
for run in RunBuilder.get_runs(params):

    device = torch.device(run.device)
    network = NetworkFactory.get_network(run.network).to(device)
    loader = DataLoader(
          train_set
        , batch_size=run.batch_size
        , num_workers=run.num_workers
    )
    optimizer = optim.Adam(network.parameters(), lr=run.lr)
    
    m.begin_run(run, network, loader)
    for epoch in range(20):
        m.begin_epoch()
        for batch in loader:
            
            images = batch[0].to(device)
            labels = batch[1].to(device)
            preds = network(images) # Pass Batch
            loss = F.cross_entropy(preds, labels) # Calculate Loss
            optimizer.zero_grad() # Zero Gradients
            loss.backward() # Calculate Gradients
            optimizer.step() # Update Weights
            
            m.track_loss(loss, batch)
            m.track_num_correct(preds, labels)
        m.end_epoch()
    m.end_run()
m.save('results')

Alright, we have the results ordered by accuracy. We can see that the 'no_max_pool' network greatly outperformed the 'max_pool' network.

In order to understand why simply removing max pooling increased our network performance so much, we need to think about the characteristics of the dataset we are training on.

To help discover clues about how max pooling impacts our data, use the Max Pooling Demo Application with some of the samples from the Fashion MNIST dataset.

Furthermore, while using the demo app, consider that our network performed two max pooling operations opposed to a single operation that is used in the app.