PyTorch torch.nn.Linear Function

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

torch.nn.Linearis a module in PyTorch used to create a fully connected layer (also called a linear layer or affine transformation).

It is one of the most basic and commonly used layers in neural networks, responsible for linearly transforming input features into output feature space.

Function Definition

torch.nn.Linear(in_features, out_features, bias=True)

Parameter Description:

  • in_features(int): The dimension of input features, i.e., the number of features output by the previous layer.
  • out_features(int): The dimension of output features, i.e., the number of features output by this layer.
  • bias(bool): Whether to add a bias term. Default isTrue. If set toFalse, then this layer will not learn bias parameters.

Attributes:

  • weight(Tensor): The learnable weight matrix with shape (out_features, in_features).
  • bias(Tensor): The learnable bias vector with shape (out_features,). Ifbias=False, then this attribute does not exist.

Mathematical Principle

nn.LinearThe calculation formula is as follows:

y = xA^T + b

where:

  • xis the input tensor, with shape (..., in_features)
  • Ais the weight matrix, with shape (out_features, in_features)
  • bis the bias vector, with shape (out_features)
  • yis the output tensor, with shape (..., out_features)

The symbol...indicates that the input can be a tensor of any dimension, and the linear transformation is applied to the last dimension.

Usage Examples

Example 1: Basic Usage

Create a simple fully connected layer that maps a 10-dimensional input to a 5-dimensional output:

Example

import torch
import torch.nn as nn

# Create a linear layer: input 10 dimensions, output 5 dimensions
linear_layer = nn.Linear(in_features=10, out_features=5, bias=True)

# Print the shapes of weight and bias
print("Weight shape:", linear_layer.weight.shape)  # torch.Size([5, 10])
print("Bias shape:", linear_layer.bias.shape)   # torch.Size([5])

# Create input tensor: batch_size=3, feature dimension=10
input_tensor = torch.randn(3, 10)

# Forward propagation
output = linear_layer(input_tensor)

print("Input shape:", input_tensor.shape)  # torch.Size([3, 10])
print("Output shape:", output.shape)        # torch.Size([3, 5])
print("Output data:\n", output)

The output result is:

权重形状: torch.Size([5, 10])
偏置形状: torch.Size([5])
输入形状: torch.Size([3, 10])
输出形状: torch.Size([3, 5])
输出数据:
 tensor([[-0.1838,  0.0607, -0.4879,  0.8981, -0.2098],
        [ 0.1513, -0.1873,  0.1866, -0.2448, -0.6012],
        [ 0.2915,  0.3053,  0.2532, -0.3372, -0.3968]],
       grad_fn=<AddmmBackward0>)

In this example, we created a fully connected layer with input 10 and output 5. The shape of the input tensor is (3, 10), where 3 is the batch size and 10 is the feature dimension. The shape of the output tensor is (3, 5).

Example 2: Without Bias

Create a linear layer without a bias term:

Example

import torch
import torch.nn as nn

# Create a linear layer without bias
linear_no_bias = nn.Linear(in_features=10, out_features=5, bias=False)

# Check whether the bias exists
print("Bias exists:", linear_no_bias.bias is None)  # True

# Forward propagation
input_tensor = torch.randn(3, 10)
output = linear_no_bias(input_tensor)

print("Output shape:", output.shape)  # torch.Size([3, 5])
print("Output:\n", output)

The output result is:

偏置是否存在: True
输出形状: torch.Size([3, 5])
输出:
 tensor([[-0.3312, -0.4113,  0.0257, -0.4876,  0.0780],
        [ 0.1513,  0.2459, -0.2983,  0.2456, -0.0727],
        [-0.0143,  0.3053,  0.1866, -0.3372,  0.2532]],
       grad_fn=<MmBackward>)

Example 3: Multidimensional Input

nn.LinearIt can handle inputs of arbitrary dimensions, and only transforms the last dimension:

Example

import torch
import torch.nn as nn

# Create a linear layer
linear = nn.Linear(in_features=10, out_features=5)

# Process 2D input (batch, features)
input_2d = torch.randn(8, 10)
output_2d = linear(input_2d)
print("2D input -> output:", input_2d.shape, "->", output_2d.shape)

# Process 3D input (batch, seq, features)
input_3d = torch.randn(4, 6, 10)
output_3d = linear(input_3d)
print("3D input -> output:", input_3d.shape, "->", output_3d.shape)

# Process 4D input (batch, channels, height, width)
input_4d = torch.randn(2, 3, 4, 10)
output_4d = linear(input_4d)
print("4D input -> output:", input_4d.shape, "->", output_4d.shape)

The output result is:

二维输入 -> 输出: torch.Size([8, 10]) -> torch.Size([8, 5])
三维输入 -> 输出: torch.Size([4, 6, 10]) -> torch.Size([4, 6, 5])
四维输入 -> 输出: torch.Size([2, 3, 4, 10]) -> torch.Size([2, 3, 4, 5])

Example 4: Use in a Neural Network

In an actual neural network,nn.Linearit is usually used in combination with other layers:

Example

import torch
import torch.nn as nn

# Define a multi-layer perceptron (MLP)
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(MLP, self).__init__()
        # First layer: input -> hidden layer
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        # Activation function
        self.relu = nn.ReLU()
        # Second layer: hidden layer -> output
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Create model
model = MLP(input_dim=784, hidden_dim=256, output_dim=10)

# Print model structure
print("Model structure:")
print(model)

# Test forward propagation
input_tensor = torch.randn(32, 784)  # batch_size=32, 28x28=784
output = model(input_tensor)

print("\nInput shape:", input_tensor.shape)  # torch.Size([32, 784])
print("Output shape:", output.shape)        # torch.Size([32, 10])

The output result is:

模型结构:
MLP(
  (fc1): Linear(in_features=784, out_features=256, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=256, out_features=10, bias=True)
)

输入形状: torch.Size([32, 784])
输出形状: torch.Size([32, 10])

Weight Initialization

By default,nn.Linearuses PyTorch's default initialization strategy.

You can also manually initialize the weights:

Example

import torch
import torch.nn as nn

# Create a linear layer
linear = nn.Linear(10, 5)

# Use Xavier initialization for weights
nn.init.xavier_uniform_(linear.weight)
# Initialize bias to zero
nn.init.zeros_(linear.bias)

# View the initialized weights
print("Weight:\n", linear.weight.data)
print("Bias:", linear.bias.data)

Difference from nn.functional.linear

PyTorch also provides a functional interfacetorch.nn.functional.linear：

Example

import torch
import torch.nn as nn
import torch.nn.functional as F

# Method 1: Using nn.Module
linear_module = nn.Linear(10, 5)
output1 = linear_module(torch.randn(3, 10))

# Method 2: Using the functional interface
weight = torch.randn(5, 10)
bias = torch.randn(5)
output2 = F.linear(torch.randn(3, 10), weight, bias)

print("nn.Module output shape:", output1.shape)
print("nn.functional output shape:", output2.shape)

The difference between the two:

  • nn.Linearis a module class that saves weight and bias parameters, making it convenient for training and saving models.
  • F.linearis a function that requires manually passing in weights and biases, often used in situations without learnable parameters.

Note: Although the mathematical operations of the two are the same, when building neural networks, we usually usenn.Linear, because it automatically registers parameters, making it convenient for optimizers to update.

Common Questions

Q1: How to view the number of parameters in a Linear layer?

For a linear layer with (in_features, out_features):

  • Number of weight parameters: in_features * out_features
  • Number of bias parameters: out_features (if bias=True)

Example

import torch
import torch.nn as nn

linear = nn.Linear(100, 50)
print("Total number of parameters:", sum(p.numel() for p in linear.parameters()))
print("Weight parameters:", linear.weight.numel())
print("Bias parameters:", linear.bias.numel())

Q2: How to freeze the parameters of a Linear layer?

If you want to fix certain layers so they do not participate in training, you can setrequires_grad=False：

Example

import torch
import torch.nn as nn

linear = nn.Linear(10, 5)

# Freeze weights so they do not participate in gradient computation
linear.weight.requires_grad = False
# Freeze bias
linear.bias.requires_grad = False

# Verify
print("Weight requires_grad:", linear.weight.requires_grad)
print("Bias requires_grad:", linear.bias.requires_grad)

Use Cases

nn.Linearis one of the most commonly used layers in neural networks. The main application scenarios include:

  • Multi-Layer Perceptron (MLP): As a fully connected layer, maps features to a new feature space.
  • Classifier: At the end of convolutional or recurrent networks, use a fully connected layer for classification.
  • Feature transformation: Apply linear transformation to data to achieve dimensionality reduction or increase.
  • Attention mechanism: In Transformer, used to generate Q, K, V matrices.

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

Other Extensions