summaryrefslogtreecommitdiff
path: root/tools/perf/util
AgeCommit message (Collapse)Author
2019-07-06perf thread: Allow references to thread objects after machine__exit()Arnaldo Carvalho de Melo
Threads are created when we either synthesize PERF_RECORD_FORK events for pre-existing threads or when we receive PERF_RECORD_FORK events from the kernel as new threads get created. We then keep them in machine->threads[].entries rb trees till when we receive a PERF_RECORD_EXIT, i.e. that thread terminated. The thread object has a reference count that is grabbed when, for instance, we keep that thread referenced in struct hist_entry, in 'perf report' and 'perf top'. When we receive a PERF_RECORD_EXIT we remove the thread object from the rb tree and move it to the corresponding machine->threads[].dead list, then we do a thread__put(), dropping the reference we had for keeping it in the rb tree. In thread__put() we were assuming that when the reference count hit zero we should remove it from the dead list by simply doing a list_del_init(&thread->node). That works well when all the thread lifetime is during the machine that has the list heads lifetime, since we know that we can do the list_del_init() and it will update the 'dead' list_head. But in 'perf sched lat' we were doing: machine__new() (via perf_session__new) process events, grabbing refcounts to keep those thread objects in 'perf sched' local data structures. machine__exit() (via perf_session__delete) which would delete the 'dead' list heads. And then doing the final thread__put() for the refcounts 'perf sched' rightfully obtained for keeping those thread object references. b00m, since thread__put() would do the list_del_init() touching a dead dead list head. Fix it by removing all the dead threads from machine->threads[].dead at machine__exit(), since whatever is there should have refcounts taken by things like 'perf sched lat', and make thread__put() check if the thread is in a linked list before removing it from that list. Reported-by: Wei Li <liwei391@huawei.com> Link: https://lkml.kernel.org/r/20190508143648.8153-1-liwei391@huawei.com Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Zhipeng Xie <xiezhipeng1@huawei.com> Link: https://lkml.kernel.org/r/20190704194355.GI10740@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-06perf header: Assign proper ff->ph in perf_event__synthesize_features()Song Liu
bpf/btf write_* functions need ff->ph->env. With this missing, pipe-mode (perf record -o -) would crash like: Program terminated with signal SIGSEGV, Segmentation fault. This patch assign proper ph value to ff. Committer testing: (gdb) run record -o - Starting program: /root/bin/perf record -o - PERFILE2 <SNIP start of perf.data headers> Thread 1 "perf" received signal SIGSEGV, Segmentation fault. __do_write_buf (size=4, buf=0x160, ff=0x7fffffff8f80) at util/header.c:126 126 memcpy(ff->buf + ff->offset, buf, size); (gdb) bt #0 __do_write_buf (size=4, buf=0x160, ff=0x7fffffff8f80) at util/header.c:126 #1 do_write (ff=ff@entry=0x7fffffff8f80, buf=buf@entry=0x160, size=4) at util/header.c:137 #2 0x00000000004eddba in write_bpf_prog_info (ff=0x7fffffff8f80, evlist=<optimized out>) at util/header.c:912 #3 0x00000000004f69d7 in perf_event__synthesize_features (tool=tool@entry=0x97cc00 <record>, session=session@entry=0x7fffe9c6d010, evlist=0x7fffe9cae010, process=process@entry=0x4435d0 <process_synthesized_event>) at util/header.c:3695 #4 0x0000000000443c79 in record__synthesize (tail=tail@entry=false, rec=0x97cc00 <record>) at builtin-record.c:1214 #5 0x0000000000444ec9 in __cmd_record (rec=0x97cc00 <record>, argv=<optimized out>, argc=0) at builtin-record.c:1435 #6 cmd_record (argc=0, argv=<optimized out>) at builtin-record.c:2450 #7 0x00000000004ae3e9 in run_builtin (p=p@entry=0x98e058 <commands+216>, argc=argc@entry=3, argv=0x7fffffffd670) at perf.c:304 #8 0x000000000042eded in handle_internal_command (argv=<optimized out>, argc=<optimized out>) at perf.c:356 #9 run_argv (argcp=<optimized out>, argv=<optimized out>) at perf.c:400 #10 main (argc=3, argv=<optimized out>) at perf.c:522 (gdb) After the patch the SEGSEGV is gone. Reported-by: David Carrillo Cisneros <davidca@fb.com> Signed-off-by: Song Liu <songliubraving@fb.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: kernel-team@fb.com Cc: stable@vger.kernel.org # v5.1+ Fixes: 606f972b1361 ("perf bpf: Save bpf_prog_info information as headers to perf.data") Link: http://lkml.kernel.org/r/20190620010453.4118689-1-songliubraving@fb.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf tools metric: Don't include duration_time in groupAndi Kleen
The Memory_BW metric generates groups including duration_time, which maps to a software event. For some reason this makes the group always not count. Always put duration_time outside a group when generating metrics. It's always the same time, so no need to group it. Signed-off-by: Andi Kleen <ak@linux.intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Link: http://lkml.kernel.org/r/20190628220737.13259-3-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf list: Avoid extra : for --raw metricsAndi Kleen
When printing the metrics raw, don't print : after the metricgroups. This helps the command line completion to complete those too. Signed-off-by: Andi Kleen <ak@linux.intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Link: http://lkml.kernel.org/r/20190628220737.13259-2-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf pmu: Support more complex PMU event aliasingJohn Garry
The jevent "Unit" field is used for uncore PMU alias definition. The form uncore_pmu_example_X is supported, where "X" is a wildcard, to support multiple instances of the same PMU in a system. Unfortunately this format not suitable for all uncore PMUs; take the Hisi DDRC uncore PMU for example, where the name is in the form hisi_scclX_ddrcY. For for current jevent parsing, we would be required to hardcode an uncore alias translation for each possible value of X. This is not scalable. Instead, add support for "Unit" field in the form "hisi_sccl,ddrc", where we can match by hisi_scclX and ddrcY. Tokens in Unit field are delimited by ','. Signed-off-by: John Garry <john.garry@huawei.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Ben Hutchings <ben@decadent.org.uk> Cc: Hendrik Brueckner <brueckner@linux.ibm.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Shaokun Zhang <zhangshaokun@hisilicon.com> Cc: Thomas Richter <tmricht@linux.ibm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: linuxarm@huawei.com Link: http://lkml.kernel.org/r/1561732552-143038-2-git-send-email-john.garry@huawei.com [ Shut up older gcc complianing about the last arg to strtok_r() being uninitialized, set that tmp to NULL ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf diff: Print the basic block cycles diffJin Yao
$ perf record -b ./div $ perf record -b ./div Following is the default perf diff output $ perf diff # Event 'cycles' # # Baseline Delta Abs Shared Object Symbol # ........ ......... ................ .................................. # 48.75% +0.33% div [.] main 8.21% -0.20% div [.] compute_flag 19.02% -0.12% libc-2.23.so [.] __random_r 16.17% -0.09% libc-2.23.so [.] __random 2.27% -0.03% div [.] rand@plt +0.02% [i915] [k] gen8_irq_handler 5.52% +0.02% libc-2.23.so [.] rand This patch creates a new computation selection 'cycles'. $ perf diff -c cycles # Event 'cycles' # # Baseline [Program Block Range] Cycles Diff Shared Object Symbol # ........ ....................................... ......................................... # 48.75% [div.c:42 -> div.c:45] 147 div [.] main 48.75% [div.c:31 -> div.c:40] 4 div [.] main 48.75% [div.c:40 -> div.c:40] 0 div [.] main 48.75% [div.c:42 -> div.c:42] 0 div [.] main 48.75% [div.c:42 -> div.c:44] 0 div [.] main 19.02% [random_r.c:357 -> random_r.c:360] 0 libc-2.23.so [.] __random_r 19.02% [random_r.c:357 -> random_r.c:373] 0 libc-2.23.so [.] __random_r 19.02% [random_r.c:357 -> random_r.c:376] 0 libc-2.23.so [.] __random_r 19.02% [random_r.c:357 -> random_r.c:380] 0 libc-2.23.so [.] __random_r 19.02% [random_r.c:357 -> random_r.c:392] 0 libc-2.23.so [.] __random_r 16.17% [random.c:288 -> random.c:291] 0 libc-2.23.so [.] __random 16.17% [random.c:288 -> random.c:291] 0 libc-2.23.so [.] __random 16.17% [random.c:288 -> random.c:295] 0 libc-2.23.so [.] __random 16.17% [random.c:288 -> random.c:297] 0 libc-2.23.so [.] __random 16.17% [random.c:291 -> random.c:291] 0 libc-2.23.so [.] __random 16.17% [random.c:293 -> random.c:293] 0 libc-2.23.so [.] __random 8.21% [div.c:22 -> div.c:22] 148 div [.] compute_flag 8.21% [div.c:22 -> div.c:25] 0 div [.] compute_flag 8.21% [div.c:27 -> div.c:28] 0 div [.] compute_flag 5.52% [rand.c:26 -> rand.c:27] 0 libc-2.23.so [.] rand 5.52% [rand.c:26 -> rand.c:28] 0 libc-2.23.so [.] rand 2.27% [rand@plt+0 -> rand@plt+0] 0 div [.] rand@plt 0.01% [entry_64.S:694 -> entry_64.S:694] 16 [vmlinux] [k] native_irq_return_iret 0.00% [fair.c:7676 -> fair.c:7665] 162 [vmlinux] [k] update_blocked_averages "[Program Block Range]" indicates the range of program basic block (start -> end). If we can find the source line it prints the source line otherwise it prints the symbol+offset instead. v4: --- Use source lines or symbol+offset to indicate the basic block. It should be easier to understand. v3: --- Cast 'struct hist_entry' to 'struct block_hist' in hist_entry__block_fprintf. Use symbol_conf.report_block to check if executing hist_entry__block_fprintf. v2: --- Keep standard perf diff format and display the 'Baseline' and 'Shared Object'. The output is sorted by "Baseline" and the basic blocks in the same function are sorted by cycles diff. Signed-off-by: Jin Yao <yao.jin@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Jin Yao <yao.jin@intel.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/1561713784-30533-7-git-send-email-yao.jin@linux.intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf diff: Use hists to manage basic blocks per symbolJin Yao
The hist__account_cycles() can account cycles per basic block. The basic block information is saved in cycles_hist structure. This patch processes each symbol, get basic blocks from cycles_hist and add the basic block entries to a new hists (in 'struct block_hist'). Using a hists is because we need to compare, sort and print the basic blocks later. v6: --- Since 'ops' argument is removed from hists__add_entry_block, update the code accordingly. No functional change. v5: --- Since now we still carry block_info in 'struct hist_entry' we don't need to use our own new/free ops for hist entries. And the block_info is released in hist_entry__delete. v3: --- 1. In v2, we put block stuffs in 'struct hist_entry', but it's not a good design. In v3, we create a new 'struct block_hist' and cast the 'struct hist_entry' to 'struct block_hist' in some places, which can avoid adding new stuffs in 'struct hist_entry'. 2. abs() -> labs(), in block_cycles_diff_cmp(). v2: --- v1 adds the basic block entries to per data-file hists but v2 adds the basic block entries to per symbol hists. That is to keep current perf-diff format. Will show the result in next patches. Signed-off-by: Jin Yao <yao.jin@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Jin Yao <yao.jin@intel.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/1561713784-30533-5-git-send-email-yao.jin@linux.intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf hists: Add block_info in hist_entryJin Yao
The block_info contains the program basic block information, i.e, contains the start address and the end address of this basic block and how much cycles it takes. We need to compare, sort and even print out the basic block by some orders, i.e. sort by cycles. For this purpose, we add block_info field to hist_entry. In order not to impact current interface, we creates a new function hists__add_entry_block. v6: --- Remove the 'ops' argument in hists__add_entry_block Signed-off-by: Jin Yao <yao.jin@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Jin Yao <yao.jin@intel.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/1561713784-30533-3-git-send-email-yao.jin@linux.intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-02perf symbol: Create block_info structureJin Yao
'perf diff' currently can only diff symbols(functions). We should expand it to diff cycles of individual programs blocks as reported by timed LBR. This would allow to identify changes in specific code accurately. We need a new structure to maintain the basic block information, such as, symbol(function), start/end address of this block, cycles. This patch creates this structure and with some ops. Signed-off-by: Jin Yao <yao.jin@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Jin Yao <yao.jin@intel.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/1561713784-30533-2-git-send-email-yao.jin@linux.intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf annotate: Add csky supportMao Han
This patch add basic arch initialization and instruction associate support for the csky CPU architecture. E.g.: $ perf annotate --stdio2 Samples: 161 of event 'cpu-clock:pppH', 4000 Hz, Event count (approx.): 40250000, [percent: local period] test_4() /usr/lib/perf-test/callchain_test Percent Disassembly of section .text: 00008420 <test_4>: test_4(): subi sp, sp, 4 st.w r8, (sp, 0x0) mov r8, sp subi sp, sp, 8 subi r3, r8, 4 movi r2, 0 st.w r2, (r3, 0x0) ↓ br 2e 100.00 14: subi r3, r8, 4 ld.w r2, (r3, 0x0) subi r3, r8, 8 st.w r2, (r3, 0x0) subi r3, r8, 4 ld.w r3, (r3, 0x0) addi r2, r3, 1 subi r3, r8, 4 st.w r2, (r3, 0x0) 2e: subi r3, r8, 4 ld.w r2, (r3, 0x0) lrw r3, 0x98967f // 8598 <main+0x28> cmplt r3, r2 ↑ bf 14 mov r0, r0 mov r0, r0 mov sp, r8 ld.w r8, (sp, 0x0) addi sp, sp, 4 ← rts Signed-off-by: Mao Han <han_mao@c-sky.com> Acked-by: Guo Ren <guoren@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: linux-csky@vger.kernel.org Link: http://lkml.kernel.org/r/d874d7782d9acdad5d98f2f5c4a6fb26fbe41c5d.1561531557.git.han_mao@c-sky.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf stat: Fix metrics with --no-mergeAndi Kleen
Since Fixes: 8c5421c016a4 ("perf pmu: Display pmu name when printing unmerged events in stat") using --no-merge adds the PMU name to the evsel name. This breaks the metric value lookup because the parser doesn't know about this. Remove the extra postfixes for the metric evaluation. Signed-off-by: Andi Kleen <ak@linux.intel.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: Agustin Vega-Frias <agustinv@codeaurora.org> Cc: Kan Liang <kan.liang@linux.intel.com> Fixes: 8c5421c016a4 ("perf pmu: Display pmu name when printing unmerged events in stat") Link: http://lkml.kernel.org/r/20190624193711.35241-5-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf stat: Fix group lookup for metric groupAndi Kleen
The metric group code tries to find a group it added earlier in the evlist. Fix the lookup to handle groups with partially overlaps correctly. When a sub string match fails and we reset the match, we have to compare the first element again. I also renamed the find_evsel function to find_evsel_group to make its purpose clearer. With the earlier changes this fixes: Before: % perf stat -M UPI,IPC sleep 1 ... 1,032,922 uops_retired.retire_slots # 1.1 UPI 1,896,096 inst_retired.any 1,896,096 inst_retired.any 1,177,254 cpu_clk_unhalted.thread After: % perf stat -M UPI,IPC sleep 1 ... 1,013,193 uops_retired.retire_slots # 1.1 UPI 932,033 inst_retired.any 932,033 inst_retired.any # 0.9 IPC 1,091,245 cpu_clk_unhalted.thread Signed-off-by: Andi Kleen <ak@linux.intel.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: Kan Liang <kan.liang@linux.intel.com> Fixes: b18f3e365019 ("perf stat: Support JSON metrics in perf stat") Link: http://lkml.kernel.org/r/20190624193711.35241-4-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf stat: Don't merge events in the same PMUAndi Kleen
Event merging is mainly to collapse similar events in lots of different duplicated PMUs. It can break metric displaying. It's possible for two metrics to have the same event, and when the two events happen in a row the second wouldn't be displayed. This would also not show the second metric. To avoid this don't merge events in the same PMU. This makes sense, if we have multiple events in the same PMU there is likely some reason for it (e.g. using multiple groups) and we better not merge them. While in theory it would be possible to construct metrics that have events with the same name in different PMU no current metrics have this problem. This is the fix for perf stat -M UPI,IPC (needs also another bug fix to completely work) Signed-off-by: Andi Kleen <ak@linux.intel.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: Kan Liang <kan.liang@linux.intel.com> Fixes: 430daf2dc7af ("perf stat: Collapse identically named events") Link: http://lkml.kernel.org/r/20190624193711.35241-3-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf stat: Make metric event lookup more robustAndi Kleen
After setting up metric groups through the event parser, the metricgroup code looks them up again in the event list. Make sure we only look up events that haven't been used by some other metric. The data structures currently cannot handle more than one metric per event. This avoids problems with multiple events partially overlapping. Signed-off-by: Andi Kleen <ak@linux.intel.com> Acked-by: Jiri Olsa <jolsa@kernel.org> Cc: Kan Liang <kan.liang@linux.intel.com> Link: http://lkml.kernel.org/r/20190624193711.35241-2-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01tools lib: Move argv_{split,free} from tools/perf/util/Arnaldo Carvalho de Melo
This came from the kernel lib/argv_split.c, so move it to tools/lib/argv_split.c, to get it closer to the kernel structure. We need to audit the usage of argv_split() to figure out if it is really necessary to do have one allocation per argv[] entry, looking at one of its users I guess that is not the case and we probably are even leaking those allocations by not using argv_free() judiciously, for later. With this we further remove stuff from tools/perf/util/, reducing the perf specific codebase and encouraging other tools/ code to use these routines so as to keep the style and constructs used with the kernel. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-j479s1ive9h75w5lfg16jroz@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf tools: Drop strxfrchar(), use strreplace() equivalent from kernelArnaldo Carvalho de Melo
No change in behaviour intended, just reducing the codebase and using something available in tools/lib/. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-oyi6zif3810nwi4uu85odnhv@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-07-01perf tools: Ditch rtrim(), use strim() from tools/libArnaldo Carvalho de Melo
Cleaning up a bit more tools/perf/util/ by using things we got from the kernel and have in tools/lib/ Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-7hluuoveryoicvkclshzjf1k@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-26perf tools: Remove trim() implementation, use tools/lib's strim()Arnaldo Carvalho de Melo
Moving more stuff out of tools/perf/util/ and using the kernel idiom. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-wpj8rktj62yse5dq6ckny6de@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-26perf tools: Ditch rtrim(), use skip_spaces() to get closer to the kernelArnaldo Carvalho de Melo
No change in behaviour, just using the same kernel idiom for such operation. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: André Goddard Rosa <andre.goddard@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-a85lkptkt0ru40irpga8yf54@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-26perf metricgroup: Use strsep()Arnaldo Carvalho de Melo
No change in behaviour intended, trivial optimization done by avoiding looking for spaces in 'g' right after setting it to "No_group". Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-f2siadtp3hb5o0l1w7bvd8bk@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-26perf strfilter: Use skip_spaces()Arnaldo Carvalho de Melo
No change in behaviour. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-p9rtamq7lvre9zhti70azfwe@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-26perf probe: Use skip_spaces() for argv handlingArnaldo Carvalho de Melo
The skip_sep() routine has the same implementation as skip_spaces(), recently adopted from the kernel, sources, switch to it. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-0ix211a81z2016dl5nmtdci4@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf time-utils: Use skip_spaces()Arnaldo Carvalho de Melo
No change in behaviour intended. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jin Yao <yao.jin@linux.intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-cpugv7qd5vzhbtvnlydo90jv@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf header: Use skip_spaces() in __write_cpudesc()Arnaldo Carvalho de Melo
No change in behaviour. Cc: Stephane Eranian <eranian@google.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-0dbfpi70aa66s6mtd8z6p391@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf stat: Use recently introduced skip_spaces()Arnaldo Carvalho de Melo
No change in behaviour. Cc: Andi Kleen <ak@linux.intel.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-ncpvp4eelf8fqhuy29uv56z9@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf tools: Use linux/ctype.h in more placesArnaldo Carvalho de Melo
There were a few places where we still were using the libc version of ctype.h, switch to the one in tools/lib/ctype.c that the rest of perf uses. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-wa4nz4kt61eze88eprk20tfd@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25tools perf: Move from sane_ctype.h obtained from git to the Linux's originalArnaldo Carvalho de Melo
We got the sane_ctype.h headers from git and kept using it so far, but since that code originally came from the kernel sources to the git sources, perhaps its better to just use the one in the kernel, so that we can leverage tools/perf/check_headers.sh to be notified when our copy gets out of sync, i.e. when fixes or goodies are added to the code we've copied. This will help with things like tools/lib/string.c where we want to have more things in common with the kernel, such as strim(), skip_spaces(), etc so as to go on removing the things that we have in tools/perf/util/ and instead using the code in the kernel, indirectly and removing things like EXPORT_SYMBOL(), etc, getting notified when fixes and improvements are made to the original code. Hopefully this also should help with reducing the difference of code hosted in tools/ to the one in the kernel proper. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-7k9868l713wqtgo01xxygn12@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf tools: Add missing util.h to pick up 'page_size' variableArnaldo Carvalho de Melo
Not to depend of getting it indirectly. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-tirjsmvu4ektw0k7lm8k9lhu@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf tools: Remove old baggage that is util/include/linux/ctype.hArnaldo Carvalho de Melo
It was just including a ../util.h that wasn't even there: $ cat tools/perf/util/include/linux/../util.h cat: tools/perf/util/include/linux/../util.h: No such file or directory $ This would make kallsyms.h get util.h somehow and then files including it would get util.h defined stuff, a mess, fix it. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-wlzwken4psiat4zvfbvaoqiw@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf symbols: We need util.h in symbol-elf.c for zfree()Arnaldo Carvalho de Melo
Continuing to untangle the headers, we're about to remove the old odd baggage that is tools/perf/util/include/linux/ctype.h. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-gapezcq3p8bzrsi96vdtq0o0@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf kallsyms: Adopt hex2u64 from tools/perf/util/util.hArnaldo Carvalho de Melo
Just removing more stuff from tools/perf/, this is mostly used in the kallsyms parsing and in places in perf where kallsyms is involved, so we get it for free there. With this we reduce a bit more util.h. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-5mc1zg0jqdwgkn8c358kaba6@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf string: Move 'dots' and 'graph_dotted_line' out of sane_ctype.hArnaldo Carvalho de Melo
Those are not in that file in the git repo, lets move it from there so that we get that sane ctype code fully isolated to allow getting it in sync either with the git sources or better with the kernel sources (include/linux/ctype.h + lib/ctype.h), that way we can use check_headers.h to get notified when changes are made in the original code so that we can cherry-pick. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-ioh5sghn3943j0rxg6lb2dgs@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf ctype: Remove now unused 'spaces' variableArnaldo Carvalho de Melo
We can left justify just fine using the 'field width' modifier in %s printf, ditch this variable. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-2td8u86mia7143lbr5ttl0kf@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf ctype: Remove unused 'graph_line' variableArnaldo Carvalho de Melo
Not being used at all anywhere. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-1e567f8tn8m4ii7dy1w9dp39@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf db-export: Export synth eventsAdrian Hunter
Synthesized events are samples but with architecture-specific data stored in sample->raw_data. They are identified by attribute type PERF_TYPE_SYNTH. Add a function to export them. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190622093248.581-6-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf intel-pt: Synthesize CBR events when last seen value changesAdrian Hunter
The first core-to-bus ratio (CBR) event will not be shown if --itrace 's' option (skip initial number of events) is used, nor if time intervals are specified that do not include the start of tracing. Change the logic to record the last CBR value seen by the user, and synthesize CBR events whenever that changes. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190622093248.581-5-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf intel-pt: Add CBR value to decoder stateAdrian Hunter
For convenience, add the core-to-bus ratio (CBR) value to the decoder state. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190622093248.581-4-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf intel-pt: Cater for CBR change in PSB+Adrian Hunter
PSB+ provides status information only so the core-to-bus ratio (CBR) in PSB+ will not have changed from its previous value. However, cater for the possibility of a another CBR change that gets caught up in the PSB+ anyway. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190622093248.581-3-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf intel-pt: Decoder to output CBR changes immediatelyAdrian Hunter
The core-to-bus ratio (CBR) provides the CPU frequency. With branches enabled, the decoder was outputting CBR changes only when there was a branch. That loses the correct time of the change if the trace is not in context (e.g. not tracing kernel space). Change to output the CBR change immediately. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190622093248.581-2-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf tools: Increase MAX_NR_CPUS and MAX_CACHESKyle Meyer
Attempting to profile 1024 or more CPUs with perf causes two errors: perf record -a [ perf record: Woken up X times to write data ] way too many cpu caches.. [ perf record: Captured and wrote X MB perf.data (X samples) ] perf report -C 1024 Error: failed to set cpu bitmap Requested CPU 1024 too large. Consider raising MAX_NR_CPUS Increasing MAX_NR_CPUS from 1024 to 2048 and redefining MAX_CACHES as MAX_NR_CPUS * 4 returns normal functionality to perf: perf record -a [ perf record: Woken up X times to write data ] [ perf record: Captured and wrote X MB perf.data (X samples) ] perf report -C 1024 ... Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/20190620193630.154025-1-meyerk@stormcage.eag.rdlabs.hpecorp.net Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf thread-stack: Eliminate code duplicating thread_stack__pop_ks()Adrian Hunter
Use new function thread_stack__pop_ks() in place of equivalent code. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190619064429.14940-3-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-25perf thread-stack: Fix thread stack return from kernel for kernel-only caseAdrian Hunter
Commit f08046cb3082 ("perf thread-stack: Represent jmps to the start of a different symbol") had the side-effect of introducing more stack entries before return from kernel space. When user space is also traced, those entries are popped before entry to user space, but when user space is not traced, they get stuck at the bottom of the stack, making the stack grow progressively larger. Fix by detecting a return-from-kernel branch type, and popping kernel addresses from the stack then. Note, the problem and fix affect the exported Call Graph / Tree but not the callindent option used by "perf script --call-trace". Example: perf-with-kcore record example -e intel_pt//k -- ls perf-with-kcore script example --itrace=bep -s ~/libexec/perf-core/scripts/python/export-to-sqlite.py example.db branches calls ~/libexec/perf-core/scripts/python/exported-sql-viewer.py example.db Menu option: Reports -> Context-Sensitive Call Graph Before: (showing Call Path column only) Call Path ▶ perf ▼ ls ▼ 12111:12111 ▶ setup_new_exec ▶ __task_pid_nr_ns ▶ perf_event_pid_type ▶ perf_event_comm_output ▶ perf_iterate_ctx ▶ perf_iterate_sb ▶ perf_event_comm ▶ __set_task_comm ▶ load_elf_binary ▶ search_binary_handler ▶ __do_execve_file.isra.41 ▶ __x64_sys_execve ▶ do_syscall_64 ▼ entry_SYSCALL_64_after_hwframe ▼ swapgs_restore_regs_and_return_to_usermode ▼ native_iret ▶ error_entry ▶ do_page_fault ▼ error_exit ▼ retint_user ▶ prepare_exit_to_usermode ▼ native_iret ▶ error_entry ▶ do_page_fault ▼ error_exit ▼ retint_user ▶ prepare_exit_to_usermode ▼ native_iret ▶ error_entry ▶ do_page_fault ▼ error_exit ▼ retint_user ▶ prepare_exit_to_usermode ▶ native_iret After: (showing Call Path column only) Call Path ▶ perf ▼ ls ▼ 12111:12111 ▶ setup_new_exec ▶ __task_pid_nr_ns ▶ perf_event_pid_type ▶ perf_event_comm_output ▶ perf_iterate_ctx ▶ perf_iterate_sb ▶ perf_event_comm ▶ __set_task_comm ▶ load_elf_binary ▶ search_binary_handler ▶ __do_execve_file.isra.41 ▶ __x64_sys_execve ▶ do_syscall_64 ▶ entry_SYSCALL_64_after_hwframe ▶ page_fault ▼ entry_SYSCALL_64 ▼ do_syscall_64 ▶ __x64_sys_brk ▶ __x64_sys_access ▶ __x64_sys_openat ▶ __x64_sys_newfstat ▶ __x64_sys_mmap ▶ __x64_sys_close ▶ __x64_sys_read ▶ __x64_sys_mprotect ▶ __x64_sys_arch_prctl ▶ __x64_sys_munmap ▶ exit_to_usermode_loop ▶ __x64_sys_set_tid_address ▶ __x64_sys_set_robust_list ▶ __x64_sys_rt_sigaction ▶ __x64_sys_rt_sigprocmask ▶ __x64_sys_prlimit64 ▶ __x64_sys_statfs ▶ __x64_sys_ioctl ▶ __x64_sys_getdents64 ▶ __x64_sys_write ▶ __x64_sys_exit_group Committer notes: The first arg to the perf-with-kcore needs to be the same for the 'record' and 'script' lines, otherwise we'll record the perf.data file and kcore_dir/ files in one directory ('example') to then try to use it from the 'bep' directory, fix the instructions above it so that both use 'example'. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: stable@vger.kernel.org Fixes: f08046cb3082 ("perf thread-stack: Represent jmps to the start of a different symbol") Link: http://lkml.kernel.org/r/20190619064429.14940-2-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-24Merge tag 'v5.2-rc6' into perf/core, to refresh branchIngo Molnar
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500Thomas Gleixner
Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation # extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 4122 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Enrico Weigelt <info@metux.net> Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 480Thomas Gleixner
Based on 1 normalized pattern(s): adapted from oprofile gplv2 support extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to add the SPDX license identifier to 1 file(s) Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Enrico Weigelt <info@metux.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081204.397687630@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 479Thomas Gleixner
Based on 1 normalized pattern(s): released under the gpl v2 based on gplv2 source code extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 1 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Enrico Weigelt <info@metux.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081204.281377867@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-17perf evsel: Make perf_evsel__name() accept a NULL argumentArnaldo Carvalho de Melo
In which case it simply returns "unknown", like when it can't figure out the evsel->name value. This makes this code more robust and fixes a problem in 'perf trace' where a NULL evsel was being passed to a routine that only used the evsel for printing its name when a invalid syscall id was passed. Reported-by: Leo Yan <leo.yan@linaro.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Link: https://lkml.kernel.org/n/tip-f30ztaasku3z935cn3ak3h53@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-17perf pmu: Fix uncore PMU alias list for ARM64John Garry
In commit 292c34c10249 ("perf pmu: Fix core PMU alias list for X86 platform"), we fixed the issue of CPU events being aliased to uncore events. Fix this same issue for ARM64, since the said commit left the (broken) behaviour untouched for ARM64. Signed-off-by: John Garry <john.garry@huawei.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Ben Hutchings <ben@decadent.org.uk> Cc: Hendrik Brueckner <brueckner@linux.ibm.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Kan Liang <kan.liang@linux.intel.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Shaokun Zhang <zhangshaokun@hisilicon.com> Cc: Thomas Richter <tmricht@linux.ibm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: linuxarm@huawei.com Cc: stable@vger.kernel.org Fixes: 292c34c10249 ("perf pmu: Fix core PMU alias list for X86 platform") Link: http://lkml.kernel.org/r/1560521283-73314-2-git-send-email-john.garry@huawei.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-17perf intel-pt: Add callchain to synthesized PEBS sampleAdrian Hunter
Like other synthesized events, if there is also an Intel PT branch trace, then a call stack can also be synthesized. Add that. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190610072803.10456-12-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2019-06-17perf intel-pt: Add memory information to synthesized PEBS sampleAdrian Hunter
Add memory information from PEBS data in the Intel PT trace to the synthesized PEBS sample. This provides sample types PERF_SAMPLE_ADDR, PERF_SAMPLE_WEIGHT, and PERF_SAMPLE_TRANSACTION, but not PERF_SAMPLE_DATA_SRC. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/20190610072803.10456-11-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>