0a6545bda2
For instance: # perf probe "vfs_getname=getname_flags:72 pathname=result->name:string" Added new event: probe:vfs_getname (on getname_flags:72 with pathname=result->name:string) You can now use it in all perf tools, such as: perf record -e probe:vfs_getname -aR sleep 1 # perf trace --failure sleep 1 0.043 ( 0.010 ms): sleep/10978 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory For reference, here are all the syscalls in this case: # perf trace sleep 1 ? ( ): sleep/10976 ... [continued]: execve()) = 0 0.027 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d04000 0.044 ( 0.010 ms): sleep/10976 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory 0.057 ( 0.006 ms): sleep/10976 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) = 3 0.064 ( 0.002 ms): sleep/10976 fstat(fd: 3, statbuf: 0x7fffac22b370) = 0 0.067 ( 0.003 ms): sleep/10976 mmap(len: 111457, prot: READ, flags: PRIVATE, fd: 3) = 0x7feec8615000 0.071 ( 0.001 ms): sleep/10976 close(fd: 3) = 0 0.080 ( 0.007 ms): sleep/10976 openat(dfd: CWD, filename: /lib64/libc.so.6, flags: CLOEXEC) = 3 0.088 ( 0.002 ms): sleep/10976 read(fd: 3, buf: 0x7fffac22b538, count: 832) = 832 0.092 ( 0.001 ms): sleep/10976 fstat(fd: 3, statbuf: 0x7fffac22b3d0) = 0 0.094 ( 0.002 ms): sleep/10976 mmap(len: 8192, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) = 0x7feec8613000 0.099 ( 0.004 ms): sleep/10976 mmap(len: 3889792, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3) = 0x7feec8057000 0.104 ( 0.007 ms): sleep/10976 mprotect(start: 0x7feec8203000, len: 2097152) = 0 0.112 ( 0.005 ms): sleep/10976 mmap(addr: 0x7feec8403000, len: 24576, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3, off: 1753088) = 0x7feec8403000 0.120 ( 0.003 ms): sleep/10976 mmap(addr: 0x7feec8409000, len: 14976, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED) = 0x7feec8409000 0.128 ( 0.001 ms): sleep/10976 close(fd: 3) = 0 0.139 ( 0.001 ms): sleep/10976 arch_prctl(option: 4098, arg2: 140663540761856) = 0 0.186 ( 0.004 ms): sleep/10976 mprotect(start: 0x7feec8403000, len: 16384, prot: READ) = 0 0.204 ( 0.003 ms): sleep/10976 mprotect(start: 0x55bdc0ec3000, len: 4096, prot: READ) = 0 0.209 ( 0.004 ms): sleep/10976 mprotect(start: 0x7feec8631000, len: 4096, prot: READ) = 0 0.214 ( 0.010 ms): sleep/10976 munmap(addr: 0x7feec8615000, len: 111457) = 0 0.269 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d04000 0.271 ( 0.002 ms): sleep/10976 brk(brk: 0x55bdc2d25000) = 0x55bdc2d25000 0.274 ( 0.001 ms): sleep/10976 brk() = 0x55bdc2d25000 0.278 ( 0.007 ms): sleep/10976 open(filename: /usr/lib/locale/locale-archive, flags: CLOEXEC) = 3 0.288 ( 0.001 ms): sleep/10976 fstat(fd: 3</usr/lib/locale/locale-archive>, statbuf: 0x7feec8408aa0) = 0 0.290 ( 0.003 ms): sleep/10976 mmap(len: 113045344, prot: READ, flags: PRIVATE, fd: 3) = 0x7feec1488000 0.297 ( 0.001 ms): sleep/10976 close(fd: 3</usr/lib/locale/locale-archive>) = 0 0.325 (1000.193 ms): sleep/10976 nanosleep(rqtp: 0x7fffac22c0b0) = 0 1000.560 ( 0.006 ms): sleep/10976 close(fd: 1) = 0 1000.573 ( 0.005 ms): sleep/10976 close(fd: 2) = 0 1000.596 ( ): sleep/10976 exit_group() # And can be done systemwide, etc, with backtraces: # perf trace --max-stack=16 --failure sleep 1 0.048 ( 0.015 ms): sleep/11092 access(filename: /etc/ld.so.preload, mode: R) = -1 ENOENT No such file or directory __access (inlined) dl_main (/usr/lib64/ld-2.26.so) # Or for some specific syscalls: # perf trace --max-stack=16 -e openat --failure cat /tmp/rien cat: /tmp/rien: No such file or directory 0.251 ( 0.012 ms): cat/11106 openat(dfd: CWD, filename: /tmp/rien) = -1 ENOENT No such file or directory __libc_open64 (inlined) main (/usr/bin/cat) __libc_start_main (/usr/lib64/libc-2.26.so) _start (/usr/bin/cat) # Look for inotify* syscalls that fail, system wide, for 2 seconds, with backtraces: # perf trace -a --max-stack=16 --failure -e inotify* sleep 2 819.165 ( 0.058 ms): gmain/1724 inotify_add_watch(fd: 8<anon_inode:inotify>, pathname: /home/acme/~, mask: 16789454) = -1 ENOENT No such file or directory __GI_inotify_add_watch (inlined) _ik_watch (/usr/lib64/libgio-2.0.so.0.5400.3) _ip_start_watching (/usr/lib64/libgio-2.0.so.0.5400.3) im_scan_missing (/usr/lib64/libgio-2.0.so.0.5400.3) g_timeout_dispatch (/usr/lib64/libglib-2.0.so.0.5400.3) g_main_context_dispatch (/usr/lib64/libglib-2.0.so.0.5400.3) g_main_context_iterate.isra.23 (/usr/lib64/libglib-2.0.so.0.5400.3) g_main_context_iteration (/usr/lib64/libglib-2.0.so.0.5400.3) glib_worker_main (/usr/lib64/libglib-2.0.so.0.5400.3) g_thread_proxy (/usr/lib64/libglib-2.0.so.0.5400.3) start_thread (/usr/lib64/libpthread-2.26.so) __GI___clone (inlined) # Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Wang Nan <wangnan0@huawei.com> Link: https://lkml.kernel.org/n/tip-8f7d3mngaxvi7tlzloz3n7cs@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
243 lines
7.5 KiB
Text
243 lines
7.5 KiB
Text
perf-trace(1)
|
|
=============
|
|
|
|
NAME
|
|
----
|
|
perf-trace - strace inspired tool
|
|
|
|
SYNOPSIS
|
|
--------
|
|
[verse]
|
|
'perf trace'
|
|
'perf trace record'
|
|
|
|
DESCRIPTION
|
|
-----------
|
|
This command will show the events associated with the target, initially
|
|
syscalls, but other system events like pagefaults, task lifetime events,
|
|
scheduling events, etc.
|
|
|
|
This is a live mode tool in addition to working with perf.data files like
|
|
the other perf tools. Files can be generated using the 'perf record' command
|
|
but the session needs to include the raw_syscalls events (-e 'raw_syscalls:*').
|
|
Alternatively, 'perf trace record' can be used as a shortcut to
|
|
automatically include the raw_syscalls events when writing events to a file.
|
|
|
|
The following options apply to perf trace; options to perf trace record are
|
|
found in the perf record man page.
|
|
|
|
OPTIONS
|
|
-------
|
|
|
|
-a::
|
|
--all-cpus::
|
|
System-wide collection from all CPUs.
|
|
|
|
-e::
|
|
--expr::
|
|
--event::
|
|
List of syscalls and other perf events (tracepoints, HW cache events,
|
|
etc) to show. Globbing is supported, e.g.: "epoll_*", "*msg*", etc.
|
|
See 'perf list' for a complete list of events.
|
|
Prefixing with ! shows all syscalls but the ones specified. You may
|
|
need to escape it.
|
|
|
|
-D msecs::
|
|
--delay msecs::
|
|
After starting the program, wait msecs before measuring. This is useful to
|
|
filter out the startup phase of the program, which is often very different.
|
|
|
|
-o::
|
|
--output=::
|
|
Output file name.
|
|
|
|
-p::
|
|
--pid=::
|
|
Record events on existing process ID (comma separated list).
|
|
|
|
-t::
|
|
--tid=::
|
|
Record events on existing thread ID (comma separated list).
|
|
|
|
-u::
|
|
--uid=::
|
|
Record events in threads owned by uid. Name or number.
|
|
|
|
-G::
|
|
--cgroup::
|
|
Record events in threads in a cgroup.
|
|
|
|
Look for cgroups to set at the /sys/fs/cgroup/perf_event directory, then
|
|
remove the /sys/fs/cgroup/perf_event/ part and try:
|
|
|
|
perf trace -G A -e sched:*switch
|
|
|
|
Will set all raw_syscalls:sys_{enter,exit}, pgfault, vfs_getname, etc
|
|
_and_ sched:sched_switch to the 'A' cgroup, while:
|
|
|
|
perf trace -e sched:*switch -G A
|
|
|
|
will only set the sched:sched_switch event to the 'A' cgroup, all the
|
|
other events (raw_syscalls:sys_{enter,exit}, etc are left "without"
|
|
a cgroup (on the root cgroup, sys wide, etc).
|
|
|
|
Multiple cgroups:
|
|
|
|
perf trace -G A -e sched:*switch -G B
|
|
|
|
the syscall ones go to the 'A' cgroup, the sched:sched_switch goes
|
|
to the 'B' cgroup.
|
|
|
|
--filter-pids=::
|
|
Filter out events for these pids and for 'trace' itself (comma separated list).
|
|
|
|
-v::
|
|
--verbose=::
|
|
Verbosity level.
|
|
|
|
--no-inherit::
|
|
Child tasks do not inherit counters.
|
|
|
|
-m::
|
|
--mmap-pages=::
|
|
Number of mmap data pages (must be a power of two) or size
|
|
specification with appended unit character - B/K/M/G. The
|
|
size is rounded up to have nearest pages power of two value.
|
|
|
|
-C::
|
|
--cpu::
|
|
Collect samples only on the list of CPUs provided. Multiple CPUs can be provided as a
|
|
comma-separated list with no space: 0,1. Ranges of CPUs are specified with -: 0-2.
|
|
In per-thread mode with inheritance mode on (default), Events are captured only when
|
|
the thread executes on the designated CPUs. Default is to monitor all CPUs.
|
|
|
|
--duration::
|
|
Show only events that had a duration greater than N.M ms.
|
|
|
|
--sched::
|
|
Accrue thread runtime and provide a summary at the end of the session.
|
|
|
|
--failure::
|
|
Show only syscalls that failed, i.e. that returned < 0.
|
|
|
|
-i::
|
|
--input::
|
|
Process events from a given perf data file.
|
|
|
|
-T::
|
|
--time::
|
|
Print full timestamp rather time relative to first sample.
|
|
|
|
--comm::
|
|
Show process COMM right beside its ID, on by default, disable with --no-comm.
|
|
|
|
-s::
|
|
--summary::
|
|
Show only a summary of syscalls by thread with min, max, and average times
|
|
(in msec) and relative stddev.
|
|
|
|
-S::
|
|
--with-summary::
|
|
Show all syscalls followed by a summary by thread with min, max, and
|
|
average times (in msec) and relative stddev.
|
|
|
|
--tool_stats::
|
|
Show tool stats such as number of times fd->pathname was discovered thru
|
|
hooking the open syscall return + vfs_getname or via reading /proc/pid/fd, etc.
|
|
|
|
-f::
|
|
--force::
|
|
Don't complain, do it.
|
|
|
|
-F=[all|min|maj]::
|
|
--pf=[all|min|maj]::
|
|
Trace pagefaults. Optionally, you can specify whether you want minor,
|
|
major or all pagefaults. Default value is maj.
|
|
|
|
--syscalls::
|
|
Trace system calls. This options is enabled by default, disable with
|
|
--no-syscalls.
|
|
|
|
--call-graph [mode,type,min[,limit],order[,key][,branch]]::
|
|
Setup and enable call-graph (stack chain/backtrace) recording.
|
|
See `--call-graph` section in perf-record and perf-report
|
|
man pages for details. The ones that are most useful in 'perf trace'
|
|
are 'dwarf' and 'lbr', where available, try: 'perf trace --call-graph dwarf'.
|
|
|
|
Using this will, for the root user, bump the value of --mmap-pages to 4
|
|
times the maximum for non-root users, based on the kernel.perf_event_mlock_kb
|
|
sysctl. This is done only if the user doesn't specify a --mmap-pages value.
|
|
|
|
--kernel-syscall-graph::
|
|
Show the kernel callchains on the syscall exit path.
|
|
|
|
--max-stack::
|
|
Set the stack depth limit when parsing the callchain, anything
|
|
beyond the specified depth will be ignored. Note that at this point
|
|
this is just about the presentation part, i.e. the kernel is still
|
|
not limiting, the overhead of callchains needs to be set via the
|
|
knobs in --call-graph dwarf.
|
|
|
|
Implies '--call-graph dwarf' when --call-graph not present on the
|
|
command line, on systems where DWARF unwinding was built in.
|
|
|
|
Default: /proc/sys/kernel/perf_event_max_stack when present for
|
|
live sessions (without --input/-i), 127 otherwise.
|
|
|
|
--min-stack::
|
|
Set the stack depth limit when parsing the callchain, anything
|
|
below the specified depth will be ignored. Disabled by default.
|
|
|
|
Implies '--call-graph dwarf' when --call-graph not present on the
|
|
command line, on systems where DWARF unwinding was built in.
|
|
|
|
--print-sample::
|
|
Print the PERF_RECORD_SAMPLE PERF_SAMPLE_ info for the
|
|
raw_syscalls:sys_{enter,exit} tracepoints, for debugging.
|
|
|
|
--proc-map-timeout::
|
|
When processing pre-existing threads /proc/XXX/mmap, it may take a long time,
|
|
because the file may be huge. A time out is needed in such cases.
|
|
This option sets the time out limit. The default value is 500 ms.
|
|
|
|
PAGEFAULTS
|
|
----------
|
|
|
|
When tracing pagefaults, the format of the trace is as follows:
|
|
|
|
<min|maj>fault [<ip.symbol>+<ip.offset>] => <addr.dso@addr.offset> (<map type><addr level>).
|
|
|
|
- min/maj indicates whether fault event is minor or major;
|
|
- ip.symbol shows symbol for instruction pointer (the code that generated the
|
|
fault); if no debug symbols available, perf trace will print raw IP;
|
|
- addr.dso shows DSO for the faulted address;
|
|
- map type is either 'd' for non-executable maps or 'x' for executable maps;
|
|
- addr level is either 'k' for kernel dso or '.' for user dso.
|
|
|
|
For symbols resolution you may need to install debugging symbols.
|
|
|
|
Please be aware that duration is currently always 0 and doesn't reflect actual
|
|
time it took for fault to be handled!
|
|
|
|
When --verbose specified, perf trace tries to print all available information
|
|
for both IP and fault address in the form of dso@symbol+offset.
|
|
|
|
EXAMPLES
|
|
--------
|
|
|
|
Trace only major pagefaults:
|
|
|
|
$ perf trace --no-syscalls -F
|
|
|
|
Trace syscalls, major and minor pagefaults:
|
|
|
|
$ perf trace -F all
|
|
|
|
1416.547 ( 0.000 ms): python/20235 majfault [CRYPTO_push_info_+0x0] => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0@0x61be0 (x.)
|
|
|
|
As you can see, there was major pagefault in python process, from
|
|
CRYPTO_push_info_ routine which faulted somewhere in libcrypto.so.
|
|
|
|
SEE ALSO
|
|
--------
|
|
linkperf:perf-record[1], linkperf:perf-script[1]
|