Transformer Architecture

The Transformer architecture is a deep learning model based on the self-attention mechanism, proposed by the Google team in a 2017 paper《Attention Is All You Need》in which it was first proposed.

Transformer has completely transformed the field of natural language processing (NLP) and has become the core foundation of modern large language models (such as GPT, BERT, etc.).

Similar to recurrent neural networks (RNNs), Transformer is designed to process sequential input data such as natural language, and is suitable for tasks like machine translation and text summarization. However, unlike RNNs, Transformer does not need to process sequences step by step; instead, it can process the entire input in parallel at once.

This chapter will take you deep into the core ideas, architecture, and working principles of Transformer, helping you grasp the fundamentals of this revolutionary technology.

Transformer Architecture Diagram

On the left is the encoder, and on the right is the decoder.

1. Input Processing (Bottom)

  • Embeddings/Projections (embedding/projection layer)
    • Function: Convert input words (or tokens) into numeric vectors (e.g., "cat" → [0.2, -0.5, 0.7…]).
    • Analogy: It's like assigning a unique "ID number" to each word, but smarter (it contains semantic information).

2. Encoder (Left)

  • Multi-Head Self-Attention (multi-head self-attention)

    • Function: Allows the model to attend to all words in the input simultaneously and compute their relationships.
    • Example: In the sentence "cat chases mouse," the model learns that "cat" and "mouse" are more strongly associated than "cat" and "chases."
    • Key point:Process all words in parallel, unlike RNN, which needs to compute one by one.

  • Norm (Layer Normalization)

    • Function: Stabilizes the training process and prevents values from becoming too large or too small (similar to adjusting the "volume" to an appropriate range).

  • Feed-Forward Network

    • Function: Further processes the representation of each word (e.g., extracting more complex features).
    • Analogy: Like performing an in-depth interpretation of the vector for "cat" and adding details (e.g., "a cat is a mammal").

3. Decoder (Right)

  • Masked Multi-Head Self-Attention

    • Function: Prevents the model from "cheating" during training (it can only see current and previous words, not future ones).
    • Example: When generating "I love __," the model can only predict the next word based on "I" and "love," and cannot know in advance that the answer is "you."

  • Multi-Head Cross-Attention

    • Function: Lets the decoder ask the encoder: "Regarding the input, what should I focus on?"
    • Scenario: In translation tasks, when the decoder generates English, it references the Chinese input processed by the encoder.

  • Norm and Feed-Forward Network

    • Similar to the encoder, it normalizes and deeply processes the decoder's representations.

4. Output (Top)

  • Linear (Linear Layer)
    • Function: Maps the decoder's output to the vocabulary (e.g., predicting that the next word is "you" with the highest probability).
    • Example: With input "I love," the model may output a probability of 80% for "you" and 10% for "eat"...

Core Idea of Transformer

Self-Attention Mechanism

The core idea of Transformer is to rely entirely on the attention mechanism (without recurrent or convolutional structures) to capture global dependencies in the input sequence, thereby achieving efficient parallel computation and stronger long-range dependency modeling.

Example

# Simplified self-attention computation example
def self_attention(query, key, value):
    scores = torch.matmul(query, key.transpose(-2, -1))  # Compute similarity
    weights = torch.softmax(scores, dim=-1)  # Convert to probability distribution
    output = torch.matmul(weights, value)  # Weighted sum
    return output

Formula:

\[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

  • \(Q\) (Query), \(K\) (Key), and \(V\) (Value) are matrices obtained by linear transformations of the input.
  • \(\sqrt{d_k}\) is a scaling factor that prevents the dot product from being too large, which would cause gradient vanishing.

Multi-Head Attention

  • Runs the self-attention mechanism multiple times in parallel (multiple "heads"), each head learning a different attention pattern, and finally concatenates the results.
  • Advantage: Capture richer contextual information (such as local/global dependencies, syntactic/semantic features).

Positional Encoding

  • Transformer itself has no recurrent or convolutional structure and cannot directly perceive sequence order, so positional information must be explicitly injected.
  • Use sinusoidal/cosine functions or learnable positional encoding:

Feed-Forward Network (FFN)

  • The representation at each position is further transformed by a two-layer fully connected network (with a nonlinear activation, such as ReLU).
  • Formula: \( \text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 \).

Residual Connection & Layer Normalization

  • Residual connection: Adds the input directly to the output (e.g., \( x + \text{Sublayer}(x) \)) to alleviate gradient vanishing.
  • Layer normalization: Normalizes the output of each layer to accelerate training.

Parallel Processing Capability

Unlike traditional RNN/LSTM, Transformer can process the entire sequence in parallel, greatly improving training efficiency.

Global Dependency Modeling

No matter how far apart words are, Transformer can directly establish dependencies between them, solving the long-range dependency problem.

Encoder-Decoder Architecture

Transformer adopts the classic encoder-decoder architecture, but each part is composed of multiple stacked identical modules.

Encoder

Component Function Description Importance
Multi-Head Attention Captures semantic information from different subspaces ★★★★★
Feed-Forward Network Nonlinear transformation and feature extraction ★★★★
Residual Connection Alleviates the vanishing gradient problem ★★★★
Layer Normalization Stabilizes the training process ★★★★

Decoder

The decoder adds the following on top of the encoder:

  1. Masked multi-head attention: Prevent the current position from attending to future information
  2. Encoder-decoder attention: Integrate source language information

Positional Encoding

Since Transformer has no recurrent structure, positional information needs to be explicitly injected.

Sinusoidal Positional Encoding Formula

The formula is divided into two parts, corresponding to even and odd dimensions of the positional encoding vector:Even dimensions (2i)，Odd dimensions (2i+1)。

Parameter description:

  • pos: The position of the word in the sequence (e.g., the 1st word, the 2nd word, etc.).

  • i: The dimension index of the positional encoding vector (0 ≤ i < d_model/2）。

  • d_model: The dimension of the positional encoding (usually the same as the word vector dimension, such as 512, 768, etc.).

Characteristics of Positional Encoding

  1. Relative position sensitivity: Can learn relative positional relationships
  2. Extensible length: Can handle sequences longer than those seen in training
  3. Determinism: No learning required, computed directly

Example

# Positional Encoding Implementation Example
class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(max_len, d_model)
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

Advantages of Transformer

Comparison with Traditional Architectures

Characteristics Transformer RNN/LSTM CNN
Parallelism Fully parallel Sequence processing Locally parallel
Long-range dependencies Direct modeling Step-by-step transmission Limited receptive field
Training speed Fast Slow Medium
Memory consumption Relatively high Medium Low

Practical Application Advantages

  1. Outstanding performance: Achieves state-of-the-art on most NLP tasks
  2. Scalability: Model scale can be easily expanded (e.g., GPT-3)
  3. Multimodal adaptation: Can be applied to various data types such as text, images, and speech
  4. Transfer learning friendly: The pretraining + fine-tuning paradigm is highly effective

Practical Exercises

Exercise 1: Implement Basic Self-Attention

Example

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, V)

Exercise 2: Understanding Positional Encoding

  1. Visualize positional encoding vectors at different positions
  2. Compute similarity between positional encodings and observe positional relationships

Frequently Asked Questions

Q: Why is Transformer more suitable for long text than RNN?A: RNN needs to transmit information step by step, and long-range dependencies are easily lost; Transformer, however, can directly establish relationships between any two words.

Q: Why is the sine function chosen for positional encoding?A: The sine function is periodic, can represent relative positional relationships, and can be extended to sequence lengths not seen during training.

Q: Where is the main computational bottleneck of Transformer?A: The space complexity of the self-attention mechanism is O(n²), which consumes a large amount of memory when processing long sequences.

Summary

The Transformer architecture has fundamentally transformed sequence modeling through the self-attention mechanism. Its core advantages are:

  1. Powerful parallel computing capability
  2. Efficient long-term dependency modeling
  3. Flexible scalability
  4. Excellent performance

Mastering the fundamentals of Transformer is an important foundation for learning modern NLP technology and is also the key to understanding cutting-edge models such as BERT and GPT.

Other Extensions