What is multi-core all about?

When single-core CPU performance improvements hit bottlenecks (power wall, frequency wall, pipeline complexity wall), engineers came up with a "simple and crude" but effective solution: placing multiple CPU cores on one chip. This lecture will introduce the principles and advantages of multi-core, as well as Amdahl's law's limits on parallel speedup.

Everyday analogy: a fast-food kitchen

Imagine a fast-food restaurant during the lunch rush.

Single-core mode: Only one chef. He is desperately trying to speed up (increase frequency) and optimize the cooking process (pipeline), but no matter how much he optimizes, only one dish can be cooked at a time. Customers wait in a long line and become impatient.

Multi-core mode: The boss hires four chefs, each with their own stove. The four chefs can cook four different dishes at the same time. There is no need for each chef to become faster, but the overall output increases severalfold.

This is the design philosophy of multi-core:When a single core is hard to speed up further, it is better to add more cores and let them work in parallel.

Multi-core does not mean "make 4 copies of everything." The 4 cores usually share the same memory, the same disk interface, and the same operating system; they only duplicate at the "compute unit" level. Just like 4 chefs sharing the same refrigerator and ingredient warehouse.

Single-core vs multi-core: architecture comparison

The dilemma of single-core CPUs

In the early 2000s, the CPU industry improved performance by constantly increasing frequency—from 1 GHz to 2 GHz to 3 GHz. But this path quickly reached its end.

Three major bottlenecks of single-core frequency:

power wall: The higher the frequency, the exponential growth in power consumption, and heat dissipation becomes a physical limit.
memory wall: The CPU is too fast, memory cannot keep up, and most of the time it is idly waiting for data.
ILP wall: Instruction-level parallelism (ILP) has been exploited to its limit; fewer and fewer instructions in a single instruction stream can be parallelized.

Around 2005, Intel and AMD successively abandoned the pure "frequency race" and turned to the multi-core route.

Structure of a multi-core CPU

A typical multi-core CPU internal structure is as follows:

  +-----------------------------------------------------------+
  |                         CPU 芯片                          |
  |  +-------------+  +-------------+  +-------------+  +-----+
  |  |   核心 1    |  |   核心 2    |  |   核心 3    |  |核 4 |
  |  | L1 Cache   |  | L1 Cache   |  | L1 Cache   |  | ... |
  |  | L2 Cache   |  | L2 Cache   |  | L2 Cache   |  |     |
  |  +------+------+  +------+------+  +------+------+  +--+--+
  |         |                |                |              |
  |         +----------------+----------------+--------------+
  |                          |
  |                    +-----+------+
  |                    | 共享 L3 Cache |
  |                    +-----+------+
  |                          |
  |                    +-----+------+
  |                    | 内存控制器   | ← 连接主内存 (RAM)
  |                    +------------+
  +-----------------------------------------------------------+

Key design:

Each core has its own L1 and L2 caches (private).
Multiple cores share one L3 cache
All cores access the same main memory through the memory controller.
The operating system is responsible for assigning tasks to different cores.

Amdahl's law: the ceiling of multi-core speedup.

The intuition that "more cores means faster" is not always right. In 1967, computer scientist Gene Amdahl proposed a famous law.

Amdahl's Law: The overall speedup of a task is limited by the portion that must be executed serially.

The formula is as follows:

加速比 = 1 / (S + P/N)

其中：
  S = 任务中必须串行执行的比例（0~1）
  P = 任务中可以并行执行的比例（0~1），且 S + P = 1
  N = 并行处理单元的数量（核心数）

This formula reveals a cruel fact:

Even with an infinite number of cores, if only 50% of the task can be parallelized, the speedup can be at most 2x.

Derivation: when N approaches infinity, P/N approaches 0, and the speedup approaches 1/S.

Parallelizable proportion	1 core	2 cores	4 cores	8 cores	16-core	Infinite cores (the upper limit)
50%	1.00x	1.33x	1.60x	1.78x	1.88x	2.00x
75%	1.00x	1.60x	2.29x	2.91x	3.37x	4.00x
90%	1.00x	1.82x	3.08x	4.71x	6.40x	10.00x
95%	1.00x	1.90x	3.48x	5.93x	9.14x	20.00x
99%	1.00x	1.98x	3.88x	7.48x	13.91x	100.00x

Observe the pattern in this table:

The higher the parallelism ratio, the more significant the multi-core speedup.A task that is 99% parallelizable can be sped up nearly 14 times on 16 cores.
More cores, smaller marginal returns.The speedup improvement from 2 cores to 4 cores is far greater than from 16 cores to 32 cores.
The serial part is the hard limitWhen only 50% is parallelizable, even with ten thousand cores, it can be at most 2 times faster.

Amdahl's Law explains why "more cores does not necessarily mean faster." If your program is mainly serial logic—for example, a computation step must wait for the result of the previous step—then no matter how many cores, they can only watch and can't help.

Interactive demo: Amdahl's Law calculator

The Python code below calculates and prints a comparison table of speedups according to Amdahl's law, and also plots how the speedup changes with the number of cores, so you can intuitively feel the impact of the "serial bottleneck."

Example

"""
Amdahl's Law Calculator (example demo)
Calculate the speedup under different parallel ratios and core counts
Print comparison table and trend analysis
"""

def amdahl_speedup(parallel_fraction, num_cores):
"""
Amdahl's Law core formula
parallel_fraction: fraction that can be executed in parallel (0.0 ~ 1.0)
num_cores: number of cores
Return speedup
"""
if num_cores == 1:
return 1.0
serial_fraction = 1.0 - parallel_fraction
speedup = 1.0 / (serial_fraction + parallel_fraction / num_cores)
return speedup

def amdahl_limit(parallel_fraction):
"""
Calculate the theoretical speedup upper limit of Amdahl's Law (as the number of cores tends to infinity)
When N→∞, P/N→0, speedup → 1/S
"""
serial_fraction = 1.0 - parallel_fraction
if serial_fraction == 0:
return float('inf') # Fully parallelizable, theoretically can be infinitely accelerated
return 1.0 / serial_fraction

def print_comparison_table(parallel_fractions, core_counts):
"""Print the complete speedup comparison table."""
# Table header
header = f"{'Cores':>6} |"
for pf in parallel_fractions:
header += f" {pf*100:>5.0f}% parallelizable |"
print("=" * (12 + 12 * len(parallel_fractions)))
print(Amdahl's Law - Multi-core Speedup Comparison Table)
print("=" * (12 + 12 * len(parallel_fractions)))
print(header)
print("-" * (12 + 12 * len(parallel_fractions)))

for cores in core_counts:
row = f" {cores:>5} |"
for pf in parallel_fractions:
speedup = amdahl_speedup(pf, cores)
row += f" {speedup:>6.2f}x |"
print(row)

# Theoretical upper limit
print("-" * (12 + 12 * len(parallel_fractions)))
limit_row = f" {'Upper limit':>5} |"
for pf in parallel_fractions:
limit = amdahl_limit(pf)
if limit == float('inf'):
limit_row += f" no upper limit |"
else:
limit_row += f" {limit:>6.2f}x |"
print(limit_row)
print("=" * (12 + 12 * len(parallel_fractions)))

def analyze_diminishing_returns(parallel_fraction, max_cores=64):
"""Analyze the diminishing marginal returns of multi-core acceleration"""
print(f"\nMarginal benefit analysis (parallelizable proportion {parallel_fraction*100:.0f}%):)
print(f{'Cores':>6} | {'Speedup':>8} | {'Improvement':>10} | {'Efficiency':>8})
print("-" * 42)

prev_speedup = 1.0
cores = 1
while cores <= max_cores:
speedup = amdahl_speedup(parallel_fraction, cores)
improvement = speedup - prev_speedup
# Efficiency = actual speedup / number of cores (100% under perfect linear speedup)
efficiency = (speedup / cores) * 100
print(f" {cores:>6} | {speedup:>8.2f}x | +{improvement:>8.3f}x | {efficiency:>7.1f}%")
prev_speedup = speedup
if cores == 1:
cores = 2
else:
cores *= 2

def compare_serial_vs_parallel(serial_work_ms, parallel_work_ms, num_cores):
"""
Use the time taken by a specific task to demonstrate Amdahl's law
serial_work_ms: time spent on the serial part (milliseconds)
parallel_work_ms: parallelizable part time (single-core, milliseconds)
num_cores: number of available cores
"""
total_single_ms = serial_work_ms + parallel_work_ms
parallel_fraction = parallel_work_ms / total_single_ms

parallel_time_multi = parallel_work_ms / num_cores
total_multi_ms = serial_work_ms + parallel_time_multi

speedup = total_single_ms / total_multi_ms

print(f"\nSimulation of specific task duration:)
print(fSerial part elapsed: {serial_work_ms} ms (cannot be accelerated))
print(fTime spent in parallelizable part: {parallel_work_ms} ms (single-core))
print(fParallel fraction: {parallel_fraction*100:.0f}%)
print(f"")
print(fSingle-core total time: {total_single_ms} ms)
print(f" {num_cores} 核total耗when: {total_multi_ms:.1f} ms（stringline {serial_work_ms} + Parallelism {parallel_work_ms}/{num_cores} = {parallel_time_multi:.1f}）")
print(f" Actual speedup: {speedup:.2f}x")
print(fAmdahl limit: {amdahl_limit(parallel_fraction):.2f}x)
return speedup

# ===== Main program =====
print("=" * 60)
print(Amdahl's Law Calculator (example))
print("=" * 60)

# 1. Basic Comparison Table
parallel_levels = [0.50, 0.75, 0.90, 0.95, 0.99]
core_list = [1, 2, 4, 8, 16, 32, 64, 128, 256]
print_comparison_table(parallel_levels, core_list)

# 2. Marginal benefit analysis (using 75% parallelizability as an example)
analyze_diminishing_returns(0.75, max_cores=64)

# 3. Specific task simulation
print("\n" + "=" * 60)
print(" Real-scenario simulation")
print("=" * 60)

# Scenario 1: Video rendering (highly parallel, 95%)
print("\n[Scene 1] Video rendering task")
print(Serial part: Read file, initialize encoder = 10ms)
print(Parallel part: per-frame rendering = 200ms (single core))
compare_serial_vs_parallel(10, 200, 16)

# Scenario 2: Database Transaction (moderately parallelizable, 50%)
print("\n[Scenario 2] Database transaction processing")
print(" Serial part: log writing, lock management = 50ms")
print(Parallel part: query execution = 50ms (single core))
compare_serial_vs_parallel(50, 50, 16)

# Scenario 3: Pure computation (highly parallel, 99%)
print("\n[Scenario 3] Scientific computing / matrix operations)
print(Serial part: result aggregation = 1ms)
print(Parallel part: matrix multiplication = 1000ms (single core))
compare_serial_vs_parallel(1, 1000, 16)

# 4. Recommended Number of Cores
print("\n" + "=" * 60)
print(Practical Advice: How to determine whether you need more cores?)
print("=" * 60)

# Calculate the 'cost-performance' inflection point at different parallel ratios
print(f" {'Parallelismratio':>10} | {'2核mention升':>10} | {'4核mention升':>10} | {'8核mention升':>10} | {'16核mention升':>9} | {'Suggestions核心number':>10}")
print("-" * 70)

for pf in [0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95, 0.99]:
ratios = []
prev = 1.0
recommendations = []
for cores in [2, 4, 8, 16]:
sp = amdahl_speedup(pf, cores)
gain = (sp / prev - 1) * 100 # Percentage improvement relative to the previous level
ratios.append(gain)
prev = sp
if gain < 10:
recommendations.append(cores)

# Suggestion: Improvement below 10% is not worth upgrading
suggested = ">=16 cores" if not recommendations else f{recommendations[0]} cores are enough

print(f" {pf*100:>9.0f}% | {ratios[0]:>9.1f}% | {ratios[1]:>9.1f}% | {ratios[2]:>9.1f}% | {ratios[3]:>8.1f}% | {suggested:>10}")

print()
print(" Rule of thumb:")
print(- Daily office work, web browsing (low parallel ratio): 4-6 cores are sufficient)
print(- Programming compilation, light gaming: 6-8 cores)
print(- Video rendering, 3D modeling: 8-16 cores)
print(- Scientific computing, AI training: the more cores the better (but also consider VRAM and memory bandwidth))

Running the above code, you will see several key phenomena:

The parallel proportion determines everything: A task that is 50% parallelizable, even with 256 cores, achieves a speedup of less than 2 times. But a 99% parallelizable task can achieve nearly 14 times speedup on 16 cores.
Diminishing marginal returns: The biggest improvement is from 1 core to 2 cores; after that, each time the core count doubles, the additional speedup becomes smaller and smaller.
Efficiency drop: When the core count doubles, the efficiency (actual speedup / number of cores) continues to decline. This is why blindly piling on cores is not cost-effective.

Interactive demo: multi-core task allocation animation

The following demonstration assigns 12 computation tasks to1 coreand4 coresbe processed. In 1-core mode, tasks are queued and executed serially; in 4-core mode, four tasks are processed in parallel at the same time. Observe the difference in progress bar fill speed and completion time between the two modes.

Multi-core task assignment animation demo (example)

Task queue

Pending task queue (total12one)

Comparison panel

Single-core panel

1 core (serial processing)

Core 1 Progress

Multi-core panel

4 cores (parallel processing)

Core 1 Progress

Core 2 Progress

Core 3 Progress

Core 4 Progress

Timer

1 core time

0.0 s

4 core time

0.0 s

Speedup

—

Result text

Button

Multi-core in real-world applications

How the operating system schedules multi-core

The scheduler in the operating system is responsible for assigning different processes and threads to different cores.

It will consider:

Load balancing: Try to keep every core busy, avoiding "one core in trouble, many cores watching."
cache affinity: try to keep the same thread running on the same core, leveraging that core's private cache
Power management: When the load is low, shut down some cores to save power.

Multi-core vs Hyper-Threading

Besides physical multi-core, there is also a technology calledHyper-Threading (SMT)。

Hyper-Threading makes one physical core look like two "logical cores." When one thread is waiting for data (cache miss), the core can switch to another thread to continue working.

Solution	Hardware cost	Performance improvement	Applicable scenarios
Physical multi-core	High (requires duplicating the entire execution unit)	Nearly double (ideal case)	Sustained high-load tasks
Hyper-threading (SMT)	Low (only requires extra registers + control logic).	About 10%~30%	Fully utilize the waiting time

Many modern CPUs use both technologies at the same time. For example, an Intel Core i7 might be "4 physical cores + hyper-threading = 8 logical cores."

Summary and verification

One-sentence summary：many核心ThroughInsingle芯片Top放putmultiple CPU 核心comeParallel executionNosameTask。But阿姆reach尔decide律指出，real际SpeedupDepends onTaskMediumcanParallelismofratio——stringlinePartdeterminealreadyadd速Upper limit，More cores, smaller marginal returns.。

self-test questions

A task has 80% that can be executed in parallel. On a 4-core CPU, what speedup does Amdahl's law give?
Answer:S=0.2, P=0.8, N=4。Speedup = 1/(0.2 + 0.8/4) = 1/(0.2 + 0.2) = 1/0.4 = 2.5x。alsoJustYes说 4 cores只带comealready 2.5 timesadd速，Efficiencyonly 62.5%。
If a task can be almost 100% parallelized, what is the theoretical upper limit? Why is it still unreachable in practice?
Answer: The theoretical upper limit is infinity (speedup = number of cores). But in practice it is limited by: (1) overhead in splitting and merging tasks; (2) latency in synchronization and communication between cores; (3) contention for shared resources (such as memory bandwidth).
Why can modern server CPUs have 64 or even 128 cores, while personal computers usually have only 4 to 16 cores?
Answer: Requests handled by servers (such as web requests and database queries) are naturally highly parallel—each user request is independent, so the parallelizable proportion is close to 100%. On the other hand, everyday tasks on personal computers (such as browser rendering and Office document processing) have limited parallelism; too many cores would be of no use and instead increase power consumption and cost.

other extensions