CNNs, RNNs and Transformers: The Main Neural Network Designs

The basic network from the earlier lessons connects every input to every neuron. That works for small tables of numbers, but images, audio and text have structure: nearby pixels belong together, and word order matters. Special network designs, called architectures, use that structure. Three matter most.

Convolutional neural networks (CNNs) for images

A convolution slides a small grid of weights, called a filter, across an image. At each position, it multiplies the filter by the pixels underneath and adds them up. A filter that responds to a vertical edge lights up wherever such an edge appears, anywhere in the image.

import numpy as np

# A tiny 6x6 grey image: dark on the left, bright on the right (a vertical edge)
image = np.array([[0, 0, 0, 9, 9, 9]] * 6, dtype=float)

# A 3x3 filter that responds to "dark on the left, bright on the right"
edge_filter = np.array([[-1, 0, 1],
                        [-1, 0, 1],
                        [-1, 0, 1]], dtype=float)

def convolve(img, k):
    """Slide the filter over the image; each output is the sum of the filter times the patch under it."""
    out = np.zeros((img.shape[0] - 2, img.shape[1] - 2))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            out[i, j] = np.sum(img[i:i + 3, j:j + 3] * k)
    return out

print(convolve(image, edge_filter).astype(int))
print("flat image gives:", convolve(np.full((6, 6), 5.0), edge_filter).max())

[[ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]]
flat image gives: 0.0

The output is large exactly where the image changes from dark to bright, and zero on flat areas. A CNN learns its filters during training. Early layers learn edges and colours; deeper layers combine them into textures, shapes and whole objects. Because the same filter is reused across the image, CNNs need far fewer weights than fully connected networks.

Recurrent neural networks (RNNs) for sequences

An RNN reads a sequence one step at a time, such as one word or one time point, and keeps a hidden state that carries information forward. Variants such as LSTMs and GRUs are better at remembering over longer spans. RNNs were the standard for text and time series, but they process steps one by one, which is slow to train, and they still struggle with very long sequences.

Transformers

Transformers use attention, which lets every position look directly at every other position, and they process all positions in parallel. That made it practical to train on huge datasets, and they now dominate text, are widely used for images and audio, and power all large language models. See attention and transformers in the LLM course.

Choosing an architecture

Data	Common choice today
Images	Pre-trained CNNs or vision transformers
Text	Pre-trained transformers
Audio and speech	Transformer-based models
Time series	Gradient boosting with lag features, or transformer and RNN models for complex cases
Tabular data	Usually gradient boosting, not deep learning

Transfer learning

Training large networks from scratch needs huge datasets and a lot of compute. Transfer learning starts from a model pre-trained on a large general dataset and adapts it to your task with a much smaller dataset, by fine-tuning some or all of its layers. It is the default approach for images, text and audio.

Key takeaway

CNNs use sliding filters to find local patterns in images; RNNs carry a memory through sequences; transformers use attention to connect all positions at once and now dominate most AI. Start from pre-trained models and adapt them. Next: PyTorch basics.