In this article we are going to talk about SPP-Net, a CNN-based model that can accept images of varying sizes so that we can completely avoid using cropping or warping mechanism. Not only that, here I will also try to implement it from scratch with PyTorch so that you can get a better understanding about the detailed architecture.

A Brief History of SPP-Net

SPP-Net was introduced in a paper titled “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition” written by He et al. back in June 2014 [1]. It is important to note that SPP itself already existed long before this paper. So, what this paper essentially proposed was SPP-Net, which is the very first model to integrate the idea of SPP on a CNN. 

CNN vs SPP-Net

The upper flowchart in Figure 2 displays the workflow of the conventional CNN model. This process requires us to perform cropping or warping to make the images in the dataset have uniform size before passing them through the model. Meanwhile, the lower flowchart in the figure shows the flow when we use SPP layer in the CNN. Here you can see that we don’t need the cropping or warping step since we already have the SPP layer placed between the convolution layers and the fully-connected layers within the network.

If we take a closer look at the CNN architecture, what basically makes the conventional CNN able to only accept a fixed-size tensor is its fully-connected layer. The convolution layers themselves can naturally accept input of varying sizes due to the shared weights across the spatial dimension of the image. For instance, if we have a conv layer with a 3×3 kernel, then we will have 9 trainable weights (or 10 if we include the bias term) regardless of the size of the image passed through the network. This property is different from a fully-connected layer, where every single element in the tensor needs to have its corresponding weight. For example, if we have an image of size 10×10, we need to flatten it into a single-dimensional tensor of length 100, and thus we will also need that same number of weights (or 101 with the bias term). This essentially means that if the shape of the input tensor changes, then we will require a different number of weights as well. This behavior makes an FC layer unsuitable for processing images of varying resolutions.

With this problem in mind, the authors of SPP-Net proposed a mechanism where we can still utilize the property of CNN in handling varying input sizes while adapting the extracted feature map to a fixed-size vector required by the fully-connected layer. This is essentially the reason why in Figure 2 they placed the SPP layer between the stack of convolutions and the FC layers.

How SPP Layer Works

Figure 3 below displays the detailed steps of how SPP-Net processes an image. In accordance with the lower flowchart in Figure 2, we initially pass the raw image through a stack of convolution layers. In one of their experiments, they used a network of 5 convolutions which they refer to as conv₁, conv₂, conv₃, conv₄ and conv₅. For the sake of simplicity, the figure below only displays the tensor produced by the last convolution, which is the one to be forwarded to the SPP layer.

Inside the SPP layer, what we do first is to divide the spatial dimension of the tensor into grids and take the maximum value within each grid — just like a typical maxpooling operation. There are 3 grid arrangements we use here: 4×4, 2×2, and 1×1, in which we will apply each of them to the same tensor. Finally, these grid cells of maximum values are flattened and concatenated, ready to be forwarded to the fully-connected part of the network.

The grid arrangements are created independently of the spatial resolution of the image. This means that we will always have the same number of grid cells regardless of the input tensor shape, which also implies that at the end of the process we will obtain a fixed-length vector. Let’s take a look at the examples in Figure 4 and 5 below to prove this.

Figure 4 above displays a case where our input feature map has the resolution of 8×8. It is mentioned in the paper that we can find the resulting feature vector length by computing Mk, where M is the number of spatial bins (the grids) and k is the number of channels of the tensor. In this case, we assume that our tensor only has a single channel, so we can simply compute M and multiply everything with 1. With the above configuration, we will get a feature vector with a length of (4×4 + 2×2 + 1×1) × 1 = 21. Now if we try to use the same configuration on different image size, say 12×8, the SPP layer will still produce the exact same vector length as it automatically adjusts the window size and the stride. You can see in Figure 5 below that the pooling window becomes rectangle.

Advantages of SPP-Net

The way SPP works discussed above allows SPP-Net to be more robust against overfitting. In fact, what makes the conventional CNN prone to overfit is the flatten operation done on the feature map produced by the last convolution layer. This is essentially because flattening destroys the spatial structure of the feature map, forcing the FC layer to learn position-specific features, which breaks the translation-invariant property that the convolutional layers provide. By using SPP, we basically still preserve the spatial information through the smaller pooling window (the 4×4 grid) while capturing the holistic view of the image through the large pooling window (the 1×1 grid). This implies that the part of the feature vector highlighted in blue in Figures 4 and 5 stores more detailed spatial information, while the one highlighted in gray stores the general information of that image.

The reason that SPP-Net is less prone to overfitting is because of the invariance of the 1D feature vector against object deformation and spatial layout. We can think of the object deformation invariance like this: suppose in the training set we got a bunch of images of a person standing — SPP allows our model to predict the person even if they are sitting since our feature vector is now less rigid to such subtle changes. Meanwhile, spatial layout invariance refers to a condition where our model is trained with images of a person located at the center of the image yet is tested on images where the person is not located exactly in the middle. SPP allows the model to predict it correctly since the pixels are now grouped into larger spatial bins, making slight shifts in the object location undetected.

Another advantage we get by using SPP-Net is that we can train the model with images of varying resolutions and aspect ratios, which again, allows the model to be even better to handle overfitting. It is explained in the paper that the authors decided to randomly resize the input image between the range of 180×180 and 224×224 after completing each epoch.

Experimental Results

Talking about the experimental results, it is shown in Figure 6 below that implementing SPP allows a model to achieve lower classification loss. Moreover, we can decrease this loss value even further by training the model with varying image scales. This trend can actually be observed in all backbone models the authors experimented with. They therefore explicitly mention in the paper that the advantages of SPP are orthogonal to specific CNN designs — meaning that we can improve the performance of any backbone model simply by attaching an SPP layer to it.

Experiments on Detection Task

Not only on classification task, but the authors also tried to use SPP-Net to perform object detection, which some of their experimental results are summarized in Figure 7 below.

Remember that back in 2014 R-CNN was the state-of-the-art model for object detection, which unfortunately was extremely slow. You can see in the table above that SPP-based approach performed 38× faster than R-CNN while maintaining the exact same mAP (59.2). We can actually make the SPP-Net even faster by performing prediction on a single scale only, which they found that it could boost the speed over 100× faster than R-CNN by slightly sacrificing the mAP.

Talking more specifically about the underlying algorithm, R-CNN is very slow because it needs to classify approximately 2000 bounding box candidates by passing each of them through the entire CNN model one by one. This process is modified by SPP-Net, where instead of passing each object candidate through the CNN, we can just pass the entire image through it and then classify the bounding box candidates from the deeper feature map using the SPP and the FC layers (see Figure 8). By doing so, we essentially avoid performing the repeating feature extraction process done by the CNN. Instead, we can now just perform the feature extraction once and do the ~2000 classifications based on the 1-dimensional feature vectors produced by the SPP layer. And so, it makes a perfect sense that SPP-based object detection method can massively outperform R-CNN in terms of the processing time. Additionally, it might be worth noting that the varying bounding box candidate sizes won’t be a problem since SPP can easily convert them into uniform-sized vectors.

SPP-Net from Scratch

As we have understood the underlying theory of SPP-Net, we can now try to implement it from scratch. And in fact, the implementation is considerably easy.

The authors used several base models in their experiments: ZF-5, Convnet-5, Overfeat-5 and Overfeat-7. In this article I’m going to implement the ZF-5, which I think is the simplest one. I’ll leave the remaining models as an exercise for you to practice. The table given in Figure 9 below shows the architectural details of these models. It is important to note that this table only provides the CNN part. According to the paper, this part will then be connected to the SPP layer followed by two FC layers and the classification layer.

The very first thing we do in the code is to import the required modules. Here we also initialize the GRIDS variable, which is used to determine the grid arrangements we are about to use in the SPP layer. In this case, I set the values to 4, 2, and 1, indicating that we will perform pooling operation within each 4×4, 2×2, and 1×1 grid cells, following the example in Figure 3.

```

Codeblock 1

import torch
import torch.nn as nn
import torch.nn.functional as F
GRIDS = [4, 2, 1]
```

SPP Layer

What we do next is to implement the SPP layer. Take a look at Codeblock 2 below to see how I do that. If you’re familiar with PyTorch models, you will find this SPP class quite unique as it does not have the __init__() method. However, this class is valid because we indeed want to define a parameterless flow. Well, this is actually not a standard convention in deep learning, but I use this term to emphasize that this module does not have any trainable weights or biases. All it does is just transform the input tensor based on the predefined grid cell arrangements.

```

Codeblock 2

class SPP(nn.Module):
def forward(self, x):
pooled_outputs = [] #(1)
for grid in GRIDS:
pooled = F.adaptive_max_pool2d(x, output_size=(grid,grid)) #(2)
print(f'after pool\t\t: {pooled.size()}')
pooled = torch.flatten(pooled, start_dim=1) #(3)
print(f'after flatten\t\t: {pooled.size()}\n')
pooled_outputs.append(pooled) #(4)
concatenated = torch.cat(pooled_outputs, dim=1) #(5)
print(f'after concatenate\t: {concatenated.size()}\n')
return concatenated
`` Inside theforward()method, what we do first is to initialize an empty list as written at line#(1). This list will then be used to store the pooling results. Next, for each grid arrangement, we will perform pooling operation usingF.adaptive_max_pool2d()(#(2)). Adaptive pooling mechanism used here is different from a standard pooling, where in this case we can just specify the output dimension we want and let the function automatically determine the kernel size and the stride. Afterwards, we flatten the pooling result (#(3)) and append it to thepooled_outputlist (#(4)) we initialized earlier. Once the iteration is complete, we then concatenate all the elements, forming a long single-dimensional tensor (#(5)`).

In order to check if our SPP class above works properly, we can test it using the following code. The x tensor I use below is essentially the one I displayed in Figure 4 (#(1)). Here I am going to pass this tensor through the SPP layer (#(2)) and see if the resulting output is correct.

```

Codeblock 3

spp = SPP()
x = torch.tensor([[[[5, 4, 4, 5, 3, 2, 6, 5],
[2, 8, 6, 3, 1, 5, 8, 1],
[4, 3, 7, 5, 2, 4, 7, 3],
[2, 1, 3, 2, 5, 5, 4, 2],
[4, 6, 7, 9, 5, 6, 6, 7],
[6, 8, 5, 9, 4, 9, 4, 1],
[3, 7, 4, 6, 8, 4, 4, 9],
[2, 4, 4, 3, 8, 2, 9, 2]]]], dtype=torch.float32) #(1)
out = spp(x) #(2)
```
And here’s what the output looks like. You can see below that our pooling operation correctly produces 4×4, 2×2, and 1×1 tensors within each iteration, which results in three 1D vectors of length 16, 4, and 1 after being flattened. Then, these feature vectors are concatenated, forming a new vector of length 21.

```

Codeblock 3 Output

after pool : torch.Size([1, 1, 4, 4])
after flatten : torch.Size([1, 16])
after pool : torch.Size([1, 1, 2, 2])
after flatten : torch.Size([1, 4])
after pool : torch.Size([1, 1, 1, 1])
after flatten : torch.Size([1, 1])
after concatenate : torch.Size([1, 21])
`` We can also print out the values inside theout` tensor as follows. You’ll be surprised that these values are exactly the same as the one in Figure 4.

```

Codeblock 4

print(out)

Codeblock 4 Output

tensor([[8., 6., 5., 8., 4., 7., 5., 7., 8., 9., 9., 7., 7., 6., 8., 9., 8., 8.,
9., 9., 9.]])
`` We can do the exact same thing to the 12×8 tensor in Figure 5. You can see in the output of the codeblock below that despite this difference in the input size, the SPP layer still produces a feature vector of the same length since we don’t change the values in theGRIDS` list.

```

Codeblock 5

spp = SPP()
x = torch.tensor([[[4, 3, 7, 5, 2, 4, 7, 3],
[2, 1, 3, 2, 5, 5, 4, 2],
[5, 4, 4, 5, 3, 2, 6, 5],
[2, 8, 6, 3, 1, 5, 8, 1],
[4, 3, 7, 5, 2, 4, 7, 3],
[2, 1, 3, 2, 5, 5, 4, 2],
[4, 6, 7, 9, 5, 6, 6, 7],
[6, 8, 5, 9, 4, 9, 4, 1],
[3, 7, 4, 6, 8, 4, 4, 9],
[2, 4, 4, 3, 8, 2, 9, 2],
[6, 8, 5, 9, 4, 9, 4, 1],
[3, 7, 4, 6, 8, 4, 4, 9]]], dtype=torch.float32)
out = spp(x)

Codeblock 5 Output

after pool : torch.Size([1, 4, 4])
after flatten : torch.Size([1, 16])
after pool : torch.Size([1, 2, 2])
after flatten : torch.Size([1, 4])
after pool : torch.Size([1, 1, 1])
after flatten : torch.Size([1, 1])
after concatenate : torch.Size([1, 21])
```
Again, if we print out the actual values contained in the resulting tensor, we can see that it has the exact same values as the one in Figure 5 (see Codeblock 6). These two test cases essentially confirm that our SPP implementation is correct, thus ready to actually be attached to the backbone model.

```

Codeblock 6

print(out)

Codeblock 6 Output

tensor([[5., 7., 5., 7., 8., 7., 5., 8., 8., 9., 9., 9., 8., 9., 9., 9., 8., 8.,
9., 9., 9.]])
```

ZF-5 Backbone with SPP

As I’ve mentioned earlier, in this demonstration I am going to use ZF-5 as the backbone model. Here I name the class ZF5_SPPNet since we want to integrate SPP into this model. The code implementation is pretty long, so I break it down into two codeblocks: 7a and 7b. Just ensure that these two codeblocks are placed inside the same notebook cell if you want to run it on your own.

```

Codeblock 7a

class ZF5_SPPNet(nn.Module):
def init(self):
super().init()

    self.relu = nn.ReLU()


    self.conv1 = nn.Conv2d(in_channels=3, 
                           out_channels=96, 
                           kernel_size=7, 
                           stride=2, 
                           padding=0)    #(1)
    self.norm1 = nn.LocalResponseNorm(size=5)
    self.pool1 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)    #(2)


    self.conv2 = nn.Conv2d(in_channels=96, 
                           out_channels=256, 
                           kernel_size=5, 
                           stride=2, 
                           padding=1)    #(3)
    self.norm2 = nn.LocalResponseNorm(size=5)
    self.pool2 = nn.MaxPool2d(kernel_size=3, stride=2, padding=0)    #(4)


    self.conv3 = nn.Conv2d(in_channels=256, 
                           out_channels=384, 
                           kernel_size=3, 
                           stride=1, 
                           padding=1)

    self.conv4 = nn.Conv2d(in_channels=384, 
                           out_channels=384, 
                           kernel_size=3, 
                           stride=1, 
                           padding=1)

    self.conv5 = nn.Conv2d(in_channels=384, 
                           out_channels=256, 
                           kernel_size=3, 
                           stride=1, 
                           padding=1)

    self.spp = SPP()    #(4)
    spp_out_size = 256 * sum([grid**2 for grid in GRIDS])    #(5)

    self.fc6 = nn.Linear(in_features=spp_out_size, out_features=4096)
    self.dropout6 = nn.Dropout(p=0.5)    #(6)

    self.fc7 = nn.Linear(in_features=4096, out_features=4096)
    self.dropout7 = nn.Dropout(p=0.5)    #(7)

    self.fc8 = nn.Linear(in_features=4096, out_features=1000)

`` Now let’s talk about the Codeblock 7a first, which contains the entireinit()` method. According to Figure 9, the first two convolution layers of ZF-5 also have a local response norm layer and a pooling layer associated with each. It might be worth noting that local response norm was the state-of-the-art normalization mechanism at that time as batch normalization layer was only introduced a year after this paper.

One thing that makes this implementation a bit challenging is the padding of the first two convolutions and their corresponding maxpooling layers. We typically set the padding to 3 for 7×7 kernel, padding 2 for 5×5 kernel, padding 1 for 3×3 kernel, and padding 0 for 1×1 kernel so that the spatial dimension of the tensor is preserved. However, when I tried to use that convention, the resulting tensor shapes do not match with the paper, so I tweaked them manually and came up with padding values that appear to be somewhat arbitrary (see lines #(1,2,3,4)). 

As the first two convolution layers are done, we can now move on to the conv3, conv4, and conv5 layers. These remaining convolutions are easy since we only need to stack them sequentially by setting the number of input and output channels according to the guide in Figure 9. In this case we don’t need to search for the padding since all of them use 3×3 kernels, hence we can just set it to 1 as we won’t change the spatial dimension until we reach the SPP layer.

The SPP layer itself is initialized at line #(4). Don’t forget to calculate the output shape of this layer because we want to use it to specify the feature vector length accepted by the subsequent FC layer (#(5)). The stack of FC layers is also straightforward to implement, where it consists of two hidden layers of 4096 neurons (fc6 and fc7). The fc7 layer is then followed by the final output layer fc8, which returns the classification tensor of length 1000, matching the number of classes available in the ImageNet 2012 dataset. Additionally, here we also initialize two dropout layers (#(6,7)) to be placed after each of the two hidden FC layers.

As the __init__() method is completed, we will now move on to the forward() method which I write in Codeblock 7b below. I am not going to explain this code though, because what we basically do here is just to connect the layers one by one.

```

Codeblock 7b

def forward(self, x):
    print(f'original\t\t: {x.size()}')

    x = self.norm1(self.relu(self.conv1(x)))
    print(f'after conv1\t\t: {x.size()}')
    x = self.pool1(x)
    print(f'after pool1\t\t: {x.size()}')

    x = self.norm2(self.relu(self.conv2(x)))
    print(f'after conv2\t\t: {x.size()}')
    x = self.pool2(x)
    print(f'after pool2\t\t: {x.size()}')

    x = self.relu(self.conv3(x))
    print(f'after conv3\t\t: {x.size()}')

    x = self.relu(self.conv4(x))
    print(f'after conv4\t\t: {x.size()}')

    x = self.relu(self.conv5(x))
    print(f'after conv5\t\t: {x.size()}\n')

    x = self.spp(x)
    print(f'after spp\t\t: {x.size()}\n')

    x = self.dropout6(self.relu(self.fc6(x)))
    print(f'after fc6\t\t: {x.size()}')

    x = self.dropout7(self.relu(self.fc7(x)))
    print(f'after fc7\t\t: {x.size()}')

    x = self.fc8(x)
    print(f'after fc8\t\t: {x.size()}')

    return x

`` Now let’s check if ourZF5_SPPNetclass works properly by running the following codeblock. Here I am trying to pass the dummy tensorx` through the network, which simulates an RGB image of size 224×224.

```

Codeblock 8

zf5sppnet = ZF5_SPPNet()
x = torch.randn(1, 3, 224, 224)
out = zf5sppnet(x)
`` You can see in the resulting output below that our 224×224 tensor becomes 55×55 after being processed by the first convolution and its corresponding maxpooling layer (#(1)). Next, the second conv-pooling stack downsamples it further to 13×13 (#(3)). If you go back to Figure 9, you will notice the table is a bit imprecise since the feature map size of 27×27 written in the *conv2* column should be the output of the second convolution (#(2)), not the output of the pooling layer (#(3)). The subsequentconv3,conv4, andconv5layers (#(4–5)) are quite simple, where what they essentially do are just changing the number of layers from 256 to 384 and back to 256 again while preserving the spatial dimension. The resulting 256×13×13 tensor is then forwarded to the SPP layer. Remember that since ourGRIDSlist has 3 values, then our SPP will produce 3 feature vectors as well, each having the length of 4096, 1024, and 256 as shown at lines#(6),#(7), and#(8). These layers are then concantenated, forming a long one-dimensional vector of length 5376 (#(9)). Afterwards, we can feed this feature vector into the FC layers until we eventually got the 1000-class prediction vector (#(10)`).

```

Codeblock 8 Output

original : torch.Size([1, 3, 224, 224])
after conv1 : torch.Size([1, 96, 109, 109])
after pool1 : torch.Size([1, 96, 55, 55]) #(1)
after conv2 : torch.Size([1, 256, 27, 27]) #(2)
after pool2 : torch.Size([1, 256, 13, 13]) #(3)
after conv3 : torch.Size([1, 384, 13, 13]) #(4)
after conv4 : torch.Size([1, 384, 13, 13])
after conv5 : torch.Size([1, 256, 13, 13]) #(5)
after pool : torch.Size([1, 256, 4, 4])
after flatten : torch.Size([1, 4096]) #(6)
after pool : torch.Size([1, 256, 2, 2])
after flatten : torch.Size([1, 1024]) #(7)
after pool : torch.Size([1, 256, 1, 1])
after flatten : torch.Size([1, 256]) #(8)
after concatenate : torch.Size([1, 5376]) #(9)
after spp : torch.Size([1, 5376])
after fc6 : torch.Size([1, 4096])
after fc7 : torch.Size([1, 4096])
after fc8 : torch.Size([1, 1000]) #(10)
```
At this point we have verified the forward pass of this network. Thus, we can say that this model is now ready to be trained.

Ending

And that’s everything about the theory and the implementation of SPP-Net. Here I challenge you to try implementing SPP layer on modern architectures like ResNet, ConvNeXt, etc. I believe you can do that pretty easily as what you need to do is just to copy-paste the SPP class in Codeblock 2 and connect it to any CNN-based architectures. You can visit my profile [3] if you need more references on implementing deep learning architectures from scratch. 

I hope you learn something new today, feel free to reach me if you spot any mistakes in my explanation or in the code. By the way you can also access the code in my GitHub repository [4]. Thanks for reading!

References

[1] Kaiming He et al. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. Arxiv. https://arxiv.org/abs/1406.4729 [Accessed October 17, 2025].

[2] Image created originally by author.

[3] Muhammad Ardi. Medium. https://medium.com/@muhammad_ardi [Accessed October 17, 2025].

[4] MuhammadArdiPutra. SPP-Net. GitHub. https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Deep%20Learning%20From%20Scratch/SPP-Net.ipynb [Accessed October 18, 2025].