| Age | Commit message (Collapse) | Author |
|---|
|
Like trace subcommand, it should be able to pass some options to control
the tracing behavior for the function graph tracer.
But some options are limited in order to maintain the internal behavior.
For example, it can limit the function call depth like below:
# perf ftrace profile --graph-opts depth=5 -- myprog
Committer testing:
root@number:~# perf ftrace profile --graph-opts thresh=1000 -- sleep 1
# Total (us) Avg (us) Max (us) Count Function
1001419.301 500709.650 1000032.000 2 x64_sys_call
1000032.000 1000032.000 1000032.000 1 __x64_sys_clock_nanosleep
1000032.000 1000032.000 1000032.000 1 common_nsleep
1000031.000 1000031.000 1000031.000 1 do_nanosleep
1000031.000 1000031.000 1000031.000 1 hrtimer_nanosleep
1000024.000 1000024.000 1000024.000 1 schedule
1387.208 1387.208 1387.208 1 __x64_sys_execve
1386.691 1386.691 1386.691 1 do_execveat_common.isra.0
1334.170 1334.170 1334.170 1 bprm_execve
1258.413 1258.413 1258.413 1 load_elf_binary
1123.068 1123.068 1123.068 1 begin_new_exec
1113.550 1113.550 1113.550 1 mmput
1109.237 1109.237 1109.237 1 exit_mmap
root@number:~# perf ftrace profile --graph-opts thresh=1200 -- sleep 1
# Total (us) Avg (us) Max (us) Count Function
1001448.204 500724.102 1000018.000 2 x64_sys_call
1000017.000 1000017.000 1000017.000 1 __x64_sys_clock_nanosleep
1000017.000 1000017.000 1000017.000 1 common_nsleep
1000017.000 1000017.000 1000017.000 1 hrtimer_nanosleep
1000016.000 1000016.000 1000016.000 1 do_nanosleep
1000012.000 1000012.000 1000012.000 1 schedule
1430.112 1430.112 1430.112 1 __x64_sys_execve
1429.581 1429.581 1429.581 1 do_execveat_common.isra.0
1376.289 1376.289 1376.289 1 bprm_execve
1301.743 1301.743 1301.743 1 load_elf_binary
root@number:~#
Reviewed-by: James Clark <james.clark@linaro.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20250107224352.1128669-2-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This patch adds a max-latency option as discussed, in case the number of
buckets is more than 22, we don't observe the setting (for now, let's
say).
By default or if 0 is passed, the value is automatically determined
based on the number of buckets, range and minimum, so that we fill all
available buffers (equivalent to the behaviour before this patch).
We now get something like this:
# perf ftrace latency --bucket-range=20 \
--min-latency 10 \
--max-latency=100 \
-T switch_mm_irqs_off -a sleep 2
# DURATION | COUNT | GRAPH |
0 - 10 us | 1731 | ################ |
10 - 30 us | 1 | |
30 - 50 us | 0 | |
50 - 70 us | 0 | |
70 - 90 us | 0 | |
90 - 100 us | 0 | |
100 - ... us | 0 | |
Note the maximum is observed also if it doesn't cover completely a full
range (the second to last range is 10us long to let the last start at
100 sharp), this looks to me more sensible and eases the computations,
since we don't need to account for the range while filling the buckets.
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Clark Williams <williams@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20241112181214.1171244-5-acme@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Things below and over will be in the first and last, outlier, buckets.
Without it:
# perf ftrace latency --use-nsec --use-bpf \
--bucket-range=200 \
-T switch_mm_irqs_off -a sleep 2
# DURATION | COUNT | GRAPH |
0 - 200 ns | 0 | |
200 - 400 ns | 44 | |
400 - 600 ns | 291 | # |
600 - 800 ns | 506 | ## |
800 - 1000 ns | 148 | |
1.00 - 1.20 us | 581 | ## |
1.20 - 1.40 us | 2199 | ########## |
1.40 - 1.60 us | 1048 | #### |
1.60 - 1.80 us | 1448 | ###### |
1.80 - 2.00 us | 1091 | ##### |
2.00 - 2.20 us | 517 | ## |
2.20 - 2.40 us | 318 | # |
2.40 - 2.60 us | 370 | # |
2.60 - 2.80 us | 271 | # |
2.80 - 3.00 us | 150 | |
3.00 - 3.20 us | 85 | |
3.20 - 3.40 us | 48 | |
3.40 - 3.60 us | 40 | |
3.60 - 3.80 us | 22 | |
3.80 - 4.00 us | 13 | |
4.00 - 4.20 us | 14 | |
4.20 - ... us | 626 | ## |
#
# perf ftrace latency --use-nsec --use-bpf \
--bucket-range=20 --min-latency=1200 \
-T switch_mm_irqs_off -a sleep 2
# DURATION | COUNT | GRAPH |
0 - 1200 ns | 1243 | ##### |
1.20 - 1.22 us | 141 | |
1.22 - 1.24 us | 202 | |
1.24 - 1.26 us | 209 | |
1.26 - 1.28 us | 219 | |
1.28 - 1.30 us | 208 | |
1.30 - 1.32 us | 245 | # |
1.32 - 1.34 us | 246 | # |
1.34 - 1.36 us | 224 | # |
1.36 - 1.38 us | 219 | |
1.38 - 1.40 us | 206 | |
1.40 - 1.42 us | 190 | |
1.42 - 1.44 us | 190 | |
1.44 - 1.46 us | 146 | |
1.46 - 1.48 us | 140 | |
1.48 - 1.50 us | 125 | |
1.50 - 1.52 us | 115 | |
1.52 - 1.54 us | 102 | |
1.54 - 1.56 us | 87 | |
1.56 - 1.58 us | 90 | |
1.58 - 1.60 us | 85 | |
1.60 - ... us | 5487 | ######################## |
#
Now we want focus on the latencies starting at 1.2us, with a finer
grained range of 20ns:
This is all on a live system, so statistically interesting, but not
narrowing down on the same numbers, so a 'perf ftrace latency record'
seems interesting to then use all on the same snapshot of latencies.
A --max-latency counterpart should come next, at first limiting the
max-latency to 20 * bucket-size, as we have a fixed buckets array with
20 + 2 entries (+ for the outliers) and thus would need to make it
larger for higher latencies.
We also may need a way to ask for not considering the out of range
values (first and last buckets) when drawing the buckets bars.
Co-developed-by: Gabriele Monaco <gmonaco@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Clark Williams <williams@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20241112181214.1171244-4-acme@kernel.org
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
In addition to showing it exponentially, using log2() to figure out the
histogram index, allow for showing it linearly:
The preexisting more, the default:
# perf ftrace latency --use-nsec --use-bpf \
-T switch_mm_irqs_off -a sleep 2
# DURATION | COUNT | GRAPH |
0 - 1 ns | 0 | |
1 - 2 ns | 0 | |
2 - 4 ns | 0 | |
4 - 8 ns | 0 | |
8 - 16 ns | 0 | |
16 - 32 ns | 0 | |
32 - 64 ns | 0 | |
64 - 128 ns | 238 | # |
128 - 256 ns | 1704 | ########## |
256 - 512 ns | 672 | ### |
512 - 1024 ns | 4458 | ########################## |
1 - 2 us | 677 | #### |
2 - 4 us | 5 | |
4 - 8 us | 0 | |
8 - 16 us | 0 | |
16 - 32 us | 0 | |
32 - 64 us | 0 | |
64 - 128 us | 0 | |
128 - 256 us | 0 | |
256 - 512 us | 0 | |
512 - 1024 us | 0 | |
1 - ... ms | 0 | |
#
The new histogram mode:
# perf ftrace latency --bucket-range=150 --use-nsec --use-bpf \
-T switch_mm_irqs_off -a sleep 2
# DURATION | COUNT | GRAPH |
0 - 1 ns | 0 | |
1 - 151 ns | 265 | # |
151 - 301 ns | 1797 | ########### |
301 - 451 ns | 258 | # |
451 - 601 ns | 289 | # |
601 - 751 ns | 2049 | ############# |
751 - 901 ns | 967 | ###### |
901 - 1051 ns | 513 | ### |
1.05 - 1.20 us | 114 | |
1.20 - 1.35 us | 559 | ### |
1.35 - 1.50 us | 189 | # |
1.50 - 1.65 us | 137 | |
1.65 - 1.80 us | 32 | |
1.80 - 1.95 us | 2 | |
1.95 - 2.10 us | 0 | |
2.10 - 2.25 us | 1 | |
2.25 - 2.40 us | 1 | |
2.40 - 2.55 us | 0 | |
2.55 - 2.70 us | 0 | |
2.70 - 2.85 us | 0 | |
2.85 - 3.00 us | 1 | |
3.00 - ... us | 4 | |
#
Co-developed-by: Gabriele Monaco <gmonaco@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Clark Williams <williams@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20241112181214.1171244-3-acme@kernel.org
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The -s/--sort option is to sort the output by given column.
$ sudo perf ftrace profile -s max sync | head
# Total (us) Avg (us) Max (us) Count Function
6301.811 6301.811 6301.811 1 __do_sys_sync
6301.328 6301.328 6301.328 1 ksys_sync
5320.300 1773.433 2858.819 3 iterate_supers
2755.875 17.012 2610.633 162 sync_fs_one_sb
2728.351 682.088 2610.413 4 ext4_sync_fs [ext4]
2603.654 2603.654 2603.654 1 jbd2_log_wait_commit [jbd2]
4750.615 593.827 2597.427 8 schedule
2164.986 26.728 2115.673 81 sync_inodes_one_sb
2143.842 26.467 2115.438 81 sync_inodes_sb
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Changbin Du <changbin.du@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: https://lore.kernel.org/lkml/20240729004127.238611-5-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The 'perf ftrace profile' command is to get function execution profiles
using function-graph tracer so that users can see the total, average,
max execution time as well as the number of invocations easily.
The following is a profile for the perf_event_open syscall.
$ sudo perf ftrace profile -G __x64_sys_perf_event_open -- \
perf stat -e cycles -C1 true 2> /dev/null | head
# Total (us) Avg (us) Max (us) Count Function
65.611 65.611 65.611 1 __x64_sys_perf_event_open
30.527 30.527 30.527 1 anon_inode_getfile
30.260 30.260 30.260 1 __anon_inode_getfile
29.700 29.700 29.700 1 alloc_file_pseudo
17.578 17.578 17.578 1 d_alloc_pseudo
17.382 17.382 17.382 1 __d_alloc
16.738 16.738 16.738 1 kmem_cache_alloc_lru
15.686 15.686 15.686 1 perf_event_alloc
14.012 7.006 11.264 2 obj_cgroup_charge
#
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Changbin Du <changbin.du@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: https://lore.kernel.org/lkml/20240729004127.238611-4-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The 'graph-tail' option is to print function name as a comment at the end.
This is useful when a large function is mixed with other functions
(possibly from different CPUs).
For example,
$ sudo perf ftrace -- perf stat true
...
1) | get_unused_fd_flags() {
1) | alloc_fd() {
1) 0.178 us | _raw_spin_lock();
1) 0.187 us | expand_files();
1) 0.169 us | _raw_spin_unlock();
1) 1.211 us | }
1) 1.503 us | }
$ sudo perf ftrace --graph-opts tail -- perf stat true
...
1) | get_unused_fd_flags() {
1) | alloc_fd() {
1) 0.099 us | _raw_spin_lock();
1) 0.083 us | expand_files();
1) 0.081 us | _raw_spin_unlock();
1) 0.601 us | } /* alloc_fd */
1) 0.751 us | } /* get_unused_fd_flags */
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Changbin Du <changbin.du@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: https://lore.kernel.org/lkml/20240729004127.238611-2-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Fix the format of unordered lists so the can wrap properly.
Signed-off-by: Changbin Du <changbin.du@huawei.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20230718085242.3090797-1-changbin.du@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Add description of 'perf ftrace latency' subcommand.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Changbin Du <changbin.du@gmail.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://lore.kernel.org/r/20220321234609.90455-2-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Option doesn't take a value, make sure the man pages agree. For example:
$ perf evlist --verbose=1
Error: option `verbose' takes no value
Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lore.kernel.org/lkml/20210226183145.1878782-1-irogers@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Same as 'perf probe -F', this patch adds filter support for the ftrace
subcommand option '-F, --funcs <[FILTER]>'.
Here is an example that only lists functions which start with 'vfs_':
$ sudo perf ftrace -F vfs_*
vfs_fadvise
vfs_fallocate
vfs_truncate
vfs_open
vfs_setpos
vfs_llseek
vfs_readf
vfs_writef
...
Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/20200904152357.6053-1-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This allows us to trace single thread instead of the whole process.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-17-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '-D/--delay' to allow us to start tracing some times
later after workload is launched.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-16-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This is to have a consistent view of all graph tracer options.
The original option '--graph-depth' is marked as deprecated.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-15-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '--graph-opts thresh' to setup trace duration
threshold for funcgraph tracer.
$ sudo ./perf ftrace -G '*' --graph-opts thresh=100
3) ! 184.060 us | } /* schedule */
3) ! 185.600 us | } /* exit_to_usermode_loop */
2) ! 225.989 us | } /* schedule_idle */
2) # 4140.051 us | } /* do_idle */
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-14-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
Sometimes we want ftrace display more and longer information about the
trace.
$ sudo perf ftrace -G '*'
2) 0.979 us | mutex_unlock();
2) 1.540 us | __fsnotify_parent();
2) 0.433 us | fsnotify();
$ sudo perf ftrace -G '*' --graph-opts verbose
14160.770883 | 0) <...>-47814 | .... | 1.289 us | mutex_unlock();
14160.770886 | 0) <...>-47814 | .... | 1.624 us | __fsnotify_parent();
14160.770887 | 0) <...>-47814 | .... | 0.636 us | fsnotify();
14160.770888 | 0) <...>-47814 | .... | 0.328 us | __sb_end_write();
14160.770888 | 0) <...>-47814 | d... | 0.430 us | fpregs_assert_state_consistent();
14160.770889 | 0) <...>-47814 | d... | | do_syscall_64() {
14160.770889 | 0) <...>-47814 | .... | | __x64_sys_close() {
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-13-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds support to display irq context info for function tracer. To do
this, just specify a '--func-opts irq-info' option.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-12-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '--graph-opts noirqs' to filter out functions executed
in irq context.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-11-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '--graph-opts nosleep-time' which allow us only to
measure on-CPU time. This option is function_graph tracer only.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-10-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds support to display call trace for function tracer. To do this,
just specify a '--func-opts call-graph' option.
Example:
$ sudo perf ftrace -T vfs_read --func-opts call-graph
iio-sensor-prox-855 [003] 6168.369657: vfs_read <-ksys_read
iio-sensor-prox-855 [003] 6168.369677: <stack trace>
=> vfs_read
=> ksys_read
=> __x64_sys_read
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe
...
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-9-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '--inherit' to allow us trace children
processes spawned by our target.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-7-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '-m/--buffer-size' to allow us set the size of per-cpu
tracing buffer.
Committer testing:
Before running with this option:
# find /sys/kernel/tracing/ -name buffer_size_kb | xargs cat
1408
1408
1408
1408
1408
1408
1408
1408
1408
#
Then, run:
# perf ftrace -m 2048K | head -10
2) | mutex_unlock() {
2) ==========> |
2) | smp_irq_work_interrupt() {
2) | irq_enter() {
2) 0.121 us | rcu_irq_enter();
2) 0.128 us | irqtime_account_irq();
2) 0.719 us | }
2) | __wake_up() {
2) | __wake_up_common_lock() {
2) 0.105 us | _raw_spin_lock_irqsave();
#
Now look at those tracefs knobs:
# find /sys/kernel/tracing/ -name buffer_size_kb | xargs cat
2048
2048
2048
2048
2048
2048
2048
2048
2048
#
This should be similar to the -m option in the other perf tools, such as
'perf record', 'perf trace', etc.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-5-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
This adds an option '-F/--funcs' to list all available functions to
trace, which is read from tracing file 'available_filter_functions'.
$ sudo ./perf ftrace -F | head
trace_initcall_finish_cb
initcall_blacklisted
do_one_initcall
do_one_initcall
trace_initcall_start_cb
run_init_process
try_to_run_init_process
match_dev_by_label
match_dev_by_uuid
rootfs_init_fs_context
$
Committer notes:
This is the same command line option and for the same purpose as in
'perf probe'.
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-3-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The '-g/-G' options have already implied function_graph tracer should be
used instead of function tracer. So we don't need extra option
'--tracer' in this case.
This patch changes the behavior as below:
- If '-g' or '-G' option is on, then function_graph tracer is used.
- If '-T' or '-N' option is on, then function tracer is used.
- The function_graph has priority over function tracer.
- The option '--tracer' only take effect if neither -g/-G nor -T/-N
is specified.
Here are some examples.
This will start tracing all functions using default tracer:
$ sudo perf ftrace
This will trace all functions using function graph tracer:
$ sudo perf ftrace -G '*'
This will trace function vfs_read using function graph tracer:
$ sudo perf ftrace -G vfs_read
This will trace function vfs_read using function tracer:
$ sudo perf ftrace -T vfs_read
Committer notes:
Using '-h -G' will tell what that option is about, so to further clarify
the above examples:
# perf ftrace -h -G
-G, --graph-funcs <func> Set graph filter on given functions
# perf ftrace -h -g
-g, --nograph-funcs <func> Set nograph filter on given functions
# perf ftrace -h -T
-T, --trace-funcs <func> trace given functions only
# perf ftrace -h -N
-N, --notrace-funcs <func> do not trace given functions
#
Signed-off-by: Changbin Du <changbin.du@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: http://lore.kernel.org/lkml/20200808023141.14227-2-changbin.du@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
I've tested to process the perf man pages with asciidoctor that is
picker than asciidoc, and it revealed minor syntax errors in some
documents. Namely, the title markers aren't aligned with the previous
line, hence asciidoctor didn't recognize as titles.
This patch corrects these markers to be processed properly.
Signed-off-by: Takashi Iwai <tiwai@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20180307105441.28512-1-tiwai@suse.de
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The -D/--graph-depth option is to set max graph depth. The following
example traces max 2-depth of page fault handler.
$ sudo perf ftrace -G __do_page_fault -D 2 -- hello
...
0) | __do_page_fault() {
0) 0.063 us | down_read_trylock();
0) 0.251 us | find_vma();
0) 5.374 us | handle_mm_fault();
0) 0.054 us | up_read();
0) 7.463 us | }
...
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: kernel-team@lge.com
Link: http://lkml.kernel.org/r/20170618142302.25390-4-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The -T/--trace-funcs and -N/--notrace-funcs options are to specify
functions to enable/disable tracing dynamically.
The -G/--graph-funcs and -g/--nograph-funcs options are to set filters
for function graph tracer.
For example, to trace fault handling functions only:
$ sudo perf ftrace -T *fault hello
0) | __do_page_fault() {
0) | handle_mm_fault() {
0) 2.117 us | __handle_mm_fault();
0) 3.627 us | }
0) 7.811 us | }
0) | __do_page_fault() {
0) | handle_mm_fault() {
0) 2.014 us | __handle_mm_fault();
0) 2.424 us | }
0) 2.951 us | }
...
To trace all functions executed in __do_page_fault:
$ sudo perf ftrace -G __do_page_fault hello
2) | __do_page_fault() {
3) 0.060 us | down_read_trylock();
3) | find_vma() {
3) 0.075 us | vmacache_find();
3) 0.053 us | vmacache_update();
3) 1.246 us | }
3) | handle_mm_fault() {
3) 0.063 us | __rcu_read_lock();
3) 0.056 us | mem_cgroup_from_task();
3) 0.057 us | __rcu_read_unlock();
3) | __handle_mm_fault() {
3) | filemap_map_pages() {
3) 0.058 us | __rcu_read_lock();
3) | alloc_set_pte() {
...
But don't want to show details in handle_mm_fault:
$ sudo perf ftrace -G __do_page_fault -g handle_mm_fault hello
3) | __do_page_fault() {
3) 0.049 us | down_read_trylock();
3) | find_vma() {
3) 0.048 us | vmacache_find();
3) 0.041 us | vmacache_update();
3) 0.680 us | }
3) 0.036 us | up_read();
3) 4.547 us | } /* __do_page_fault */
...
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: kernel-team@lge.com
Link: http://lkml.kernel.org/r/20170618142302.25390-3-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The -a/--all-cpus and -C/--cpu option is for controlling tracing cpus.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: kernel-team@lge.com
Link: http://lkml.kernel.org/r/20170224011251.14946-3-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The -p (--pid) option enables to trace existing process by its pid.
Committer notes:
Testing it:
Using the function_graph tracer on a process that is just waiting for user
input and thus will make 'perf ftrace' sit there waiting for that, then press
any key on that mutt session and see what happens:
# perf ftrace -t function_graph -p `pidof mutt` | head -40
2) 1.038 us | switch_mm_irqs_off();
------------------------------------------
2) <idle>-0 => mutt-3595
------------------------------------------
2) | finish_task_switch() {
2) | smp_irq_work_interrupt() {
2) | irq_enter() {
2) 0.180 us | rcu_irq_enter();
2) 1.248 us | }
2) | __wake_up() {
2) 0.126 us | _raw_spin_lock_irqsave();
2) | __wake_up_common() {
2) | pollwake() {
2) | default_wake_function() {
2) | try_to_wake_up() {
2) 0.662 us | _raw_spin_lock_irqsave();
2) | select_task_rq_fair() {
2) 1.719 us | effective_load.isra.41();
2) 1.343 us | effective_load.isra.41();
2) | select_idle_sibling() {
2) 0.331 us | idle_cpu();
2) 1.458 us | }
2) 8.350 us | }
2) 0.200 us | _raw_spin_lock();
2) | ttwu_do_activate() {
2) | activate_task() {
2) 0.136 us | update_rq_clock.part.77();
2) | enqueue_task_fair() {
2) | enqueue_entity() {
2) 0.146 us | update_curr();
2) 0.330 us | account_entity_enqueue();
2) 0.280 us | update_cfs_shares();
2) 0.321 us | place_entity();
2) 0.206 us | __enqueue_entity();
2) 6.926 us | }
2) | enqueue_entity() {
2) 0.105 us | update_curr();
2) 0.175 us | account_entity_enqueue();
2) 0.531 us | update_cfs_shares();
#
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: kernel-team@lge.com
Link: http://lkml.kernel.org/r/20170224011251.14946-1-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
|
The 'perf ftrace' command is a simple wrapper of kernel's ftrace
functionality. It only supports single thread tracing currently and
just reads trace_pipe in text and then write it to stdout.
Committer notes:
Testing it:
# perf ftrace -f function_graph usleep 123456
<SNIP>
2) | SyS_nanosleep() {
2) | _copy_from_user() {
<SNIP>
2) 0.900 us | }
2) 1.354 us | }
2) | hrtimer_nanosleep() {
2) 0.062 us | __hrtimer_init();
2) | do_nanosleep() {
2) | hrtimer_start_range_ns() {
<SNIP>
2) 5.025 us | }
2) | schedule() {
2) 0.125 us | rcu_note_context_switch();
2) 0.057 us | _raw_spin_lock();
2) | deactivate_task() {
2) 0.369 us | update_rq_clock.part.77();
2) | dequeue_task_fair() {
<SNIP>
2) + 22.453 us | }
2) + 23.736 us | }
2) | pick_next_task_fair() {
<SNIP>
2) + 47.167 us | }
2) | pick_next_task_idle() {
<SNIP>
2) 4.462 us | }
------------------------------------------
2) usleep-20387 => <idle>-0
------------------------------------------
2) 0.806 us | switch_mm_irqs_off();
------------------------------------------
2) <idle>-0 => usleep-20387
------------------------------------------
2) 0.151 us | finish_task_switch();
2) @ 123597.2 us | }
2) 0.037 us | _cond_resched();
2) | hrtimer_try_to_cancel() {
2) 0.064 us | hrtimer_active();
2) 0.353 us | }
2) @ 123605.3 us | }
2) @ 123606.2 us | }
2) @ 123608.3 us | } /* SyS_nanosleep */
2) | __do_page_fault() {
<SNIP>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremy Eder <jeder@redhat.com>
Cc: Jiri Olsa <jolsa@redhat.com>,
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Link: http://lkml.kernel.org/n/tip-r1hgmsj4dxny8arn3o9mw512@git.kernel.org
[ Various foward port fixes, add man page ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
