Linux perf 子系统由两大核心部分构成：用户态 perf 工具用于交互，内核态 perf-event 子系统进行事件管理、调度、采样、缓冲区管理。
Table of contents
Open Table of contents
1. 使用示例
用户态 perf 工具的源码位于 tools/perf ，需要自行编译后放入 rootfs 中。
编写一个用户态程序 perf-test：
#include <stdint.h>
__attribute__((noinline))
static void foo(void)
{
volatile uint64_t x = 0;
for (uint64_t i = 0; i < 800000000ULL; i++)
x += i;
}
__attribute__((noinline))
static void bar(void)
{
volatile uint64_t x = 0;
for (uint64_t i = 0; i < 200000000ULL; i++)
x += i;
}
int main(void)
{
foo();
bar();
return 0;
}
使用 perf list 可以看到支持的事件类型，主要有 tracepoint / hardware / software / tool 四类，使用 -e 指定：
perf list
alarmtimer:alarmtimer_cancel [Tracepoint event]
alarmtimer:alarmtimer_fired [Tracepoint event]
alarmtimer:alarmtimer_start [Tracepoint event]
bpf_test_run:bpf_test_finish [Tracepoint event]
bpf_test_run:bpf_trigger_tp [Tracepoint event]
......
legacy hardware:
cpu-cycles
[Total cycles. Be wary of what happens during CPU frequency scaling
[This event is an alias of cycles]. Unit: armv8_pmuv3]
cycles
[Total cycles. Be wary of what happens during CPU frequency scaling
[This event is an alias of cpu-cycles]. Unit: armv8_pmuv3]
idle-cycles-backend
[Stalled cycles during retirement [This event is an alias of
stalled-cycles-backend]. Unit: armv8_pmuv3]
idle-cycles-frontend
[Stalled cycles during issue [This event is an alias of
stalled-cycles-frontend]. Unit: armv8_pmuv3]
stalled-cycles-backend
[Stalled cycles during retirement [This event is an alias of
idle-cycles-backend]. Unit: armv8_pmuv3]
stalled-cycles-frontend
[Stalled cycles during issue [This event is an alias of
idle-cycles-frontend]. Unit: armv8_pmuv3]
software:
alignment-faults
[Number of kernel handled memory alignment faults. Unit: software]
bpf-output
[An event used by BPF programs to write to the perf ring buffer. Unit:
software]
cgroup-switches
[Number of context switches to a task in a different cgroup. Unit:
software]
context-switches
[Number of context switches [This event is an alias of cs]. Unit:
software]
cpu-clock
[Per-CPU high-resolution timer based event. Unit: software]
cpu-migrations
[Number of times a process has migrated to a new CPU [This event is an
alias of migrations]. Unit: software]
cs
[Number of context switches [This event is an alias of
context-switches]. Unit: software]
dummy
[A placeholder event that doesn't count anything. Unit: software]
emulation-faults
[Number of kernel handled unimplemented instruction faults handled
through emulation. Unit: software]
faults
[Number of page faults [This event is an alias of page-faults]. Unit:
software]
major-faults
[Number of major page faults. Major faults require I/O to handle. Unit:
software]
migrations
[Number of times a process has migrated to a new CPU [This event is an
alias of cpu-migrations]. Unit: software]
minor-faults
[Number of minor page faults. Minor faults don't require I/O to handle.
Unit: software]
page-faults
[Number of page faults [This event is an alias of faults]. Unit:
software]
task-clock
[Per-task high-resolution timer based event. Unit: software]
tool:
core_wide
[1 if not SMT,if SMT are events being gathered on all SMT threads 1
otherwise 0. Unit: tool]
duration_time
[Wall clock interval time in nanoseconds. Unit: tool]
has_pmem
[1 if persistent memory installed otherwise 0. Unit: tool]
num_cores
[Number of cores. A core consists of 1 or more thread,with each thread
being associated with a logical Linux CPU. Unit: tool]
num_cpus
[Number of logical Linux CPUs. There may be multiple such CPUs on a
core. Unit: tool]
num_cpus_online
[Number of online logical Linux CPUs. There may be multiple such CPUs
on a core. Unit: tool]
num_dies
[Number of dies. Each die has 1 or more cores. Unit: tool]
num_packages
[Number of packages. Each package has 1 or more die. Unit: tool]
slots
[Number of functional units that in parallel can execute parts of an
instruction. Unit: tool]
smt_on
[1 if simultaneous multithreading (aka hyperthreading) is enable
otherwise 0. Unit: tool]
system_time
[System/kernel time in nanoseconds. Unit: tool]
target_cpu
[1 if CPUs being analyzed,0 if threads/processes. Unit: tool]
user_time
[User (non-kernel) time in nanoseconds. Unit: tool]
rNNN [Raw event descriptor]
armv8_pmuv3/threshold_count,event=0..0xffff,long,.../modifier[Raw event descriptor]
[(see 'man perf-list' or 'man perf-record' on how to encode it)]
breakpoint//modifier [Raw event descriptor]
kprobe/retprobe/modifier [Raw event descriptor]
software//modifier [Raw event descriptor]
tool//modifier [Raw event descriptor]
tracepoint//modifier [Raw event descriptor]
uprobe/ref_ctr_offset=0..0xffffffff,retprobe/modifier[Raw event descriptor]
mem:<addr>[/len][:access]此外还有 4 组根据多个 events 综合计算得出的 Metric 可以选择，使用 -M 指定：
Metric Groups:
Default:
CPUs_utilized
[Average CPU utilization]
backend_cycles_idle
[Backend stalls per cycle]
branch_frequency
[Branches per CPU second]
branch_miss_rate
[Branch miss rate]
cs_per_second
[Context switches per CPU second]
cycles_frequency
[Cycles per CPU second]
frontend_cycles_idle
[Frontend stalls per cycle]
insn_per_cycle
[Instructions Per Cycle]
migrations_per_second
[Process migrations to a new CPU per CPU second]
page_faults_per_second
[Page faults per CPU second]
stalled_cycles_per_instruction
[Max front or backend stalls per instruction]
Default2:
l1d_miss_rate
[L1D miss rate]
llc_miss_rate
[LLC miss rate]
Default3:
dtlb_miss_rate
[dTLB miss rate]
itlb_miss_rate
[iTLB miss rate]
l1i_miss_rate
[L1I miss rate]
Default4:
l1_prefetch_miss_rate
[L1 prefetch miss rate]1.1 stat
计数模式：
perf stat -e cycles,task-clock,sched:sched_switch,duration_time perf-test
Performance counter stats for 'perf-test':
2835640627 cycles
2835.89 msec task-clock
2 sched:sched_switch
0 duration_time
2.838354112 seconds time elapsed
2.830176000 seconds user
0.007983000 seconds sys
perf stat -M cycles_frequency perf-test
Performance counter stats for 'perf-test':
2791872947 cpu-cycles # 1.0 GHz cycles_frequency
2791.99 msec task-clock
2.793761216 seconds time elapsed
2.794174000 seconds user
0.000000000 seconds sys
1.2 record
采样模式，默认输出到perf.data，可以使用 -o 参数指定，-g 开启graph模式，-F 指定采样频率：
perf record -o test.data perf-test
perf report -i test.data
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 13K of event 'armv8_pmuv3/cycles/P'
# Event count (approx.): 3204211015
#
# Overhead Command Shared Object Symbol
# ........ ......... ..................... .............................
#
79.99% perf-test perf-test [.] 0x00000000000007e0
19.88% perf-test perf-test [.] 0x0000000000000824
0.01% perf-test [kernel.kallsyms] [k] handle_softirqs
0.01% perf-test [kernel.kallsyms] [k] el0_ia
0.01% perf-test [kernel.kallsyms] [k] kvfree_call_rcu
0.01% perf-test libc.so.6 [.] 0x000000000015b158
0.01% perf-test [kernel.kallsyms] [k] perf_iterate_ctx
0.01% perf-test ld-linux-aarch64.so.1 [.] 0x0000000000009480
0.01% perf-test [kernel.kallsyms] [k] el0_da
0.01% perf-test ld-linux-aarch64.so.1 [.] 0x0000000000020560
0.01% perf-test [kernel.kallsyms] [k] debug_smp_processor_id
0.01% perf-test [kernel.kallsyms] [k] do_dentry_open
0.01% perf-test [kernel.kallsyms] [k] do_mmap
0.01% perf-test ld-linux-aarch64.so.1 [.] 0x00000000000197b8
0.01% perf-test [kernel.kallsyms] [k] kfree
0.01% perf-test [kernel.kallsyms] [k] __pi_memset_generic
0.01% perf-test [kernel.kallsyms] [k] __pi_clear_page
0.01% perf-test [kernel.kallsyms] [k] check_preemption_disabled
0.01% perf-test [kernel.kallsyms] [k] __pcs_replace_full_main
0.01% perf-test [kernel.kallsyms] [k] __free_frozen_pages
0.00% perf-test [kernel.kallsyms] [k] kmem_cache_free
0.00% perf-test [kernel.kallsyms] [k] __zap_vma_range
0.00% perf-test [kernel.kallsyms] [k] __pte_offset_map
0.00% perf-test [kernel.kallsyms] [k] get_random_u64
0.00% perf-test [kernel.kallsyms] [k] _raw_spin_lock
0.00% perf-exec [kernel.kallsyms] [k] sized_strscpy
2. 原理分析
perf-event 将各种类型的事件源都抽象为 PMU。
在 PMU 的结构体定义中包含了一系列回调：
struct pmu {
int (*event_init)(struct perf_event *event);
int (*add)(struct perf_event *event, int flags);
void (*del)(struct perf_event *event, int flags);
void (*start)(struct perf_event *event, int flags);
void (*stop)(struct perf_event *event, int flags);
void (*read)(struct perf_event *event);
void (*enable)(struct pmu *pmu);
void (*disable)(struct pmu *pmu);
...
};2.1 初始化 event
当用户态发起一次追踪时：
perf stat -e cycles ./test实际上是构造了一个 struct perf_event_attr 通过系统调用 perf_event_open 传递给内核：
struct perf_event_attr attr = {
.type = PERF_TYPE_HARDWARE,
.config = PERF_COUNT_HW_CPU_CYCLES,
...
};来到内核后，首先通过 perf_event_alloc 构造一个struct perf_event：
perf_event_alloc(...)
{
event = alloc();
/*
* 1. 初始化 perf core 通用状态
*/
event->attr = *attr;
event->cpu = cpu;
event->state = PERF_EVENT_STATE_INACTIVE;
/*
* 2. 初始化 hw_perf_event 通用部分
*/
hwc = &event->hw;
hwc->sample_period = attr->sample_period;
...
/*
* 3. 选择 PMU + 调用 pmu->event_init()
*/
pmu = perf_init_event(event);
/*
* 此时：
*
* event->pmu = pmu
* event->hw 已被具体 PMU 初始化
*/
/*
* 4. 初始化各种附加资源
*/
...
/*
* 5. 挂到 PMU 的 event 总表
*/
list_add(&event->pmu_list, &pmu->events);
return event;
}接下来会将 event 安装到某个 task 的 perf_event_ctxp 字段上
static void perf_install_in_context(struct perf_event_context *ctx,
struct perf_event *event,
int cpu)
{
struct task_struct *task = ctx->task;
event->ctx = ctx;
/*
* 1. disabled event，而且 context 里已经有 event：
* 不需要立刻打 IPI 去重编程硬件，直接挂进去。
*/
if (event_is_off(event) && ctx->nr_events) {
lock(ctx);
add_event_to_ctx(event, ctx);
unlock(ctx);
return;
}
/*
* 2. per-CPU event：
* 必须让目标 CPU 自己执行安装。
*/
if (!task) {
cpu_function_call(cpu, __perf_install_in_context, event);
return;
}
/*
* 3. per-task event：
* 如果 task 正在某个 CPU 上运行，
* 让那个 CPU 执行 __perf_install_in_context()。
*/
if (task_is_running(task)) {
task_function_call(task, __perf_install_in_context, event);
return;
}
/*
* 4. task 当前没运行：
* 直接把 event 挂到 ctx。
* 以后 task schedule in 时再把 event 放进 PMU。
*/
lock(ctx);
add_event_to_ctx(event, ctx);
unlock(ctx);
}2.2 开启 / 停止计数
当挂载了 perf_event_ctxp 的 task 被调度执行时会来到：
static int event_sched_in(struct perf_event *event,
struct perf_event_context *ctx)
{
/*
* OFF/ERROR 状态的 event 不能被调度到 PMU 上。
* 只有 INACTIVE event 才有资格变成 ACTIVE。
*/
if (event->state <= PERF_EVENT_STATE_OFF)
return 0;
/*
* 记录这个 event 当前运行在哪个 CPU，
* 并先标记为 ACTIVE。
*/
event->oncpu = smp_processor_id();
event->state = PERF_EVENT_STATE_ACTIVE;
/*
* 修改 PMU 配置前，先临时暂停整个 PMU。
*/
perf_pmu_disable(event->pmu);
/*
* 真正把 event 加到 PMU。
*
* PERF_EF_START 表示：
* add 成功后直接启动 counter。
*
* 对硬件 PMU 来说，这里通常会：
* 1. 分配硬件 counter
* 2. 设置 event->hw.idx
* 3. 配置硬件事件
* 4. 启动 counter
*/
if (event->pmu->add(event, PERF_EF_START)) {
/*
* PMU 资源不足或 add 失败，
* 回滚到“已 enable 但没在硬件上运行”的状态。
*/
event->state = PERF_EVENT_STATE_INACTIVE;
event->oncpu = -1;
perf_pmu_enable(event->pmu);
return -EAGAIN;
}
/*
* PMU 配置完成，恢复整个 PMU。
*/
perf_pmu_enable(event->pmu);
return 0;
}当 task 被调度出去时会执行：
static void event_sched_out(struct perf_event *event)
{
/*
* 当前 event 已经不再运行在 CPU 的 PMU 上。
*/
event->state = PERF_EVENT_STATE_INACTIVE;
event->oncpu = -1;
/*
* 修改 PMU 前先暂停整个 PMU。
*/
perf_pmu_disable(event->pmu);
/*
* 从 PMU 移除 event。
*
* 对硬件 PMU 来说，del() 通常会：
* 1. stop counter
* 2. 更新 event->count
* 3. 释放硬件 counter
*/
event->pmu->del(event, 0);
/*
* PMU 配置完成后恢复 PMU。
*/
perf_pmu_enable(event->pmu);
}2.3 获取结果
计数模式：事件发生后累计到 event->count，用户态通过 read(perf_event_fd) 读取；如果 event 仍是 ACTIVE，内核会先通过 pmu->read() 把最新硬件值同步到 event->count。
采样模式：
- HW-PMU：根据
sample_period给硬件 counter 预装一个初值，让它在经过指定数量的事件后发生真实硬件 overflow；PMU 中断处理里调用 perf 的 overflow/sample 路径，把PERF_RECORD_SAMPLE写入 ring buffer。 - SW-PMU：没有真实硬件 overflow。对于
context-switches、page-faults这类事件，每发生一次对应软件事件就推进一次软件计数，达到sample_period时视为一次 overflow，然后采样并写 ring buffer；cpu-clock/task-clock这类则主要靠 timer 驱动。
2.4 多路复用
硬件 PMU 的多路复用，就是当需要监控的硬件事件数量超过可用 PMU counter 数量时，perf 会让不同 event 分批、轮流占用这些硬件 counter：当前被装入 PMU 的 event 处于 ACTIVE，其余保持 INACTIVE，轮换时通过 event_sched_out() -> pmu->del()/stop() 保存计数，再通过 event_sched_in() -> pmu->add()/start() 换入下一批；同时 perf 为每个 event 记录 time_enabled 和 time_running，最后按实际运行时间比例对计数值进行 scaling，从而用有限的硬件 counter 近似同时监控更多事件。