PyTorch torch.nn.ELU Function

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

torch.nn.ELUIt is the exponential linear unit activation function in PyTorch.

It uses an exponential function for negative values, allowing non-zero outputs for negative values and smoother gradients.

Function Definition

torch.nn.ELU(alpha=1.0, inplace=False)

Formula

ELU(x) = x, x > 0
ELU(x) = alpha * (e^x - 1), x <= 0

Usage Examples

Example 1: Basic Usage

Example

import torch
import torch.nn as nn

elu = nn.ELU(alpha=1.0)

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
output = elu(x)

print(Input:, x.tolist())
print(Output:, output.tolist())

Example 2: Comparison with ReLU

Example

import torch
import torch.nn as nn
import numpy as np

x = np.array([-3.0, -2.0, -1.0, 0.0, 1.0, 2.0])
x_t = torch.tensor(x)

print("x        ELU       ReLU")
for i in range(0, 6, 1):
    print(f"{x[i]:6.1f} {nn.ELU()(x_t[i:i+1]).item():9.4f} {nn.ReLU()(x_t[i:i+1]).item():9.4f}")

Example 3: alpha Parameter

Example

import torch
import torch.nn as nn

# Different alpha values
x = torch.tensor([-1.0])

for alpha in [0.5, 1.0, 1.5, 2.0]:
    out = nn.ELU(alpha=alpha)(x)
    print(f"alpha={alpha}: {out.item():.4f}")

Use Cases

  • Autoencoders
  • Need negative outputs
  • Smooth gradients

Note: ELU is slower to compute than ReLU, but converges faster.

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

Other Extensions