Linux perf Command

Linux 命令大全Linux Command Reference

perf is a Linux system performance analysis toolset, whose full name is Performance Event Counters. It is based on the perf_events subsystem of the Linux kernel and can provide performance analysis capabilities at both the hardware and software levels.

The main functions of perf include:

  • CPU performance analysis
  • Function call tracing
  • Hardware event statistics
  • Software event monitoring
  • System call tracing

perf Basic Syntax

The basic syntax format of the perf command is:

perf [--version] [--help] COMMAND [ARGS]

Common subcommands include:

  • stat: Performance counter statistics
  • record: Record performance data
  • report: Analyze recorded data
  • top: Real-time performance monitoring
  • list: List available events
  • annotate: Source code level analysis

Detailed Explanation of Common perf Subcommands

perf stat

Count various hardware and software events during command execution.

perf stat [options] command [command-options]

Common options:

  • -e: Specify the event to monitor
  • -p: Monitor the specified process ID
  • -a: Monitor all CPUs
  • -r: Repeat the run and display the average
  • -d: Display more detailed events

Example:

Example

# Count the execution of the ls command
perf stat ls

# Monitor a specified process
perf stat -p 1234

# Monitor specific events
perf stat -e cycles,instructions,cache-misses ls

perf record

Record performance data to a file (default: perf.data).

perf record [options] command [command-options]

Common options:

  • -g: Record call graph
  • -F: Sampling frequency (Hz)
  • -p: Record for a specified process
  • -o: Specify output file
  • -e: Specify the events to record

Example:

Example

# Record the execution of the ls command
perf record ls

# Record process 1234 at 99Hz frequency
perf record -F 99 -p 1234 -g

perf report

Analyze the data recorded by perf record.

perf report [options]

Common options:

  • -i: Specify input file
  • -n: Display sample count
  • --stdio: Text mode output
  • -g: Display call graph
  • -s: Sort by the specified field

Example:

Example

# Analyze the default perf.data file
perf report

# Analyze the specified file and output in text mode
perf report -i perf.data.old --stdio

perf top

Display the most resource-consuming functions in the system in real time.

perf top [options]

Common options:

  • -e: Specify monitoring event
  • -p: Monitor a specified process
  • -K: Hide kernel symbols
  • -U: Hide user-space symbols
  • -g: Display call graph

Example:

Example

# Monitor system performance in real time
perf top

# Monitor a specific event
perf top -e cache-misses

perf list

List all monitorable events.

perf list [hw|sw|cache|tracepoint|pmu|event_glob]

Example:

Example

# List all events
perf list

# List hardware cache events
perf list cache

perf Event Types

perf can monitor multiple types of events:

Event Type Description Example
Hardware CPU hardware events cycles, instructions
Software Kernel software events context-switches, page-faults
Cache Cache-related events cache-references, cache-misses
Tracepoints Kernel static tracepoints syscalls, block, sched
PMU Processor-specific events (vendor specific)
Breakpoints Breakpoint events mem:[:access]

perf Practical Application Examples

1. Analyze Program Performance Bottlenecks

Example

# Record program execution
perf record -g ./my_program

# Analyze the results
perf report -g

2. Find CPU Hotspot Functions

perf top -p $(pidof my_program)

3. Compare Performance Differences Between Two Runs

Example

# First run
perf record -o perf.data.1 ./my_program input1

# Second run
perf record -o perf.data.2 ./my_program input2

# Compare the differences
perf diff perf.data.1 perf.data.2

4. Analyze System Calls

Example

# List available system call tracepoints
perf list 'syscalls:*'

# Trace the open system call
perf stat -e 'syscalls:sys_enter_open' -a sleep 10

perf Usage Tips

  1. Reduce overhead: For long-running performance analysis, appropriately reduce the sampling frequency (e.g., -F 99)

  2. Symbol resolution: Ensure debug symbols are available; you can install debug packages or compile the program with -g

  3. Flame graph generation: Combine with the FlameGraph tool to generate intuitive performance analysis graphs

  4. Multi-core analysis: Use the -a option to monitor all CPUs, or use -C to specify specific CPUs

  5. User/kernel space separation: Use the -k and -u options to analyze kernel and user space respectively

Troubleshooting

  1. Permission issues：

    echo -1 > /proc/sys/kernel/perf_event_paranoid

    or run perf with sudo

  2. Missing symbol information：

    • Ensure the program is compiled with the -g option
    • Install debug symbol packages

  3. Sampling data too large：

    • Reduce sampling frequency
    • Shorten sampling time
    • Use --no-call-graph to reduce call graph information

  4. Unable to parse high-level languages such as Java/Python：

    • Requires language-specific perf tool support
    • Consider using language-specific analysis tools

Advanced Learning Resources

  1. Official documentation: The tools/perf/Documentation directory in the Linux kernel source code

  2. Book recommendations：

    • 《Systems Performance: Enterprise and the Cloud》
    • "Systems Performance: Enterprise and the Cloud"

  3. Online resources：

    • Brendan Gregg's blog and toolset
    • perf-tools toolset

  4. Related tools：

    • FlameGraph: Visualize performance data
    • bpftrace: More flexible dynamic tracing

By mastering the perf command, you can deeply analyze the performance characteristics of Linux systems, quickly locate performance bottlenecks, and optimize applications and system configurations.

Linux 命令大全Linux Command Reference

Other Extensions