This blog post is not about tracing program system calls, but about programmatically extracting all system calls and their accepted arguments in your current system. If you want to read more about tracing system calls, here’s my tutorial using strace for troubleshooting a real life scenario: Troubleshooting Linux SSH Login Delay - Why does logging in always take 10 seconds?1
I am building the next version of my 0x.tools xcapture tool for continuous OS-wide profiling of process activity in Linux. The new version is based on eBPF and uses Python as its frontend for formatting the output and storing history. One of the things xcapture collects is samples of system calls invoked by all threads and their associated stack traces (and it does so much more)!
I wanted the ability to not just report system call names but also system call argument names when relevant. As far as I know and searched, the eBPF BCC toolset doesn’t do that and there are no existing tools that easily list the system call arguments on your system.
You could parse the various C-header files and compile a list yourself, but the problem is that every platform and major kernel version has a different set of system calls defined. Some kernel build time config options can cause your current kernel to have a different set of system calls. Also system call internal numbering differs across platforms, so parsing header files is not an easy option! That’s what ausyscall2 does and currently they don’t accept additional platforms as it’s too much work.
System Call Tracepoints in Tracefs
I looked for other options and Linux tracefs is the answer! Both debugfs and tracefs have already been enabled by default on all major distros that I have worked with, but you can manually mount it yourself too.
$ mount | grep debug
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime,seclabel)
Since every Linux system call has a static kernel tracepoint defined for it, there will be a separate tracefs directory for each syscall’s entry- and exit-points:
$ sudo tree /sys/kernel/debug/tracing/events/syscalls/sys_enter_pread64 /sys/kernel/debug/tracing/events/syscalls/sys_enter_pread64 ├── enable ├── filter ├── format ├── hist ├── id ├── inject └── trigger
Each directory has a file called format describing the info provided by the specific tracepoint, let’s take a look:
$ sudo cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_pread64/format name: sys_enter_pread64 ID: 762 format: field:unsigned short common_type; offset:0; size:2; signed:0; field:unsigned char common_flags; offset:2; size:1; signed:0; field:unsigned char common_preempt_count; offset:3; size:1; signed:0; field:int common_pid; offset:4; size:4; signed:1; field:int __syscall_nr; offset:8; size:4; signed:1; field:unsigned int fd; offset:16; size:8; signed:0; field:char * buf; offset:24; size:8; signed:0; field:size_t count; offset:32; size:8; signed:0; field:loff_t pos; offset:40; size:8; signed:0; print fmt: "fd: 0x%08lx, buf: 0x%08lx, count: 0x%08lx, pos: 0x%08lx", ((unsigned long)(REC->fd)), ((unsigned long)(REC->buf)), ((unsigned long)(REC->count)), ((unsigned long)(REC->pos))
Indeed, this file provides not only syscall argument names, but also their datatype! The pread64 syscall, for example, has 4 arguments that it uses (fd, buf, count, pos). The maximum number of arguments for any syscall is 6 (all passed to the kernel syscall handler via CPU registers on x86_64 architecture).
Syscallargs Tool
As the syscall metadata of your current kernel is there, I wrote a bit of python code to parse and format this info. I use that code for xcapture, but it makes sense to publish this as a standalone tool too.
Here are its command line options and example output:
$ syscallargs -h
usage: syscallargs [-h] [-l] [-t] [-V] [--path PATH]
List kernel system calls and their arguments from Linux debugfs.
optional arguments:
-h, --help show this help message and exit
-l, --newlines Print each system call and its arguments on a new line
-t, --typeinfo Include type information for syscall arguments in the output
-V, --version Show the program version and exit
--path PATH Path to the debugfs syscalls directory:
/sys/kernel/debug/tracing/events/syscalls
The default option is to list all system calls with only their argument names, one syscall per output line. This is from Oracle Linux 8 (RHEL 8 clone) running on ARM64. Due to the default debugfs/tracefs permissions, you need to run this command as root or remount your tracefs filesystem with relaxed permissions:
$ sudo syscallargs | head socket(family, type, protocol); socketpair(family, type, protocol, usockvec); bind(fd, umyaddr, addrlen); listen(fd, backlog); accept4(fd, upeer_sockaddr, upeer_addrlen, flags); accept(fd, upeer_sockaddr, upeer_addrlen); connect(fd, uservaddr, addrlen); getsockname(fd, usockaddr, usockaddr_len); getpeername(fd, usockaddr, usockaddr_len); sendto(fd, buff, len, flags, addr, addr_len);
This output is from Ubuntu 24.04 running on an X86_64 platform. I have edited the output to show only a few lines of output below:
$ sudo syscallargs ... io_getevents(ctx_id, min_nr, nr, events, timeout); io_cancel(ctx_id, iocb, result); io_submit(ctx_id, nr, iocbpp); io_destroy(ctx); io_setup(nr_events, ctxp); flock(fd, cmd); open_by_handle_at(mountdirfd, handle, flags); name_to_handle_at(dfd, name, handle, mnt_id, flag); ...
When you use the -t option, syscallargs will also print out the type information of the syscall arguments, very useful!
$ sudo syscallargs -t ... io_getevents((aio_context_t) ctx_id, (long) min_nr, (long) nr, (struct io_event *) events, (struct __kernel_timespec *) timeout); io_cancel((aio_context_t) ctx_id, (struct iocb *) iocb, (struct io_event *) result); io_submit((aio_context_t) ctx_id, (long) nr, (struct iocb * *) iocbpp); io_destroy((aio_context_t) ctx); io_setup((unsigned) nr_events, (aio_context_t *) ctxp); flock((unsigned int) fd, (unsigned int) cmd); open_by_handle_at((int) mountdirfd, (struct file_handle *) handle, (int) flags); name_to_handle_at((int) dfd, (const char *) name, (struct file_handle *) handle, (int *) mnt_id, (int) flag); ...
There’s also the -l option if you want to print out each argument on its own line, this format might be easier for human consumption:
$ sudo syscallargs -l
socket
0: family
1: type
2: protocol
socketpair
0: family
1: type
2: protocol
3: usockvec
bind
0: fd
1: umyaddr
2: addrlen
listen
0: fd
1: backlog
And you can combine both options for full output:
$ sudo syscallargs -lt
copy_file_range
0: (int) fd_in
1: (loff_t *) off_in
2: (int) fd_out
3: (loff_t *) off_out
4: (size_t) len
5: (unsigned int) flags
sendfile64
0: (int) out_fd
1: (int) in_fd
2: (loff_t *) offset
3: (size_t) count
pwritev2
0: (unsigned long) fd
1: (const struct iovec *) vec
2: (unsigned long) vlen
3: (unsigned long) pos_l
4: (unsigned long) pos_h
5: (rwf_t) flags
Summary
The syscallarg tool allows you to quickly and programmatically look up system calls and their argument names/types on your current running kernel. It is available in the 0x.tools Github repo. Let me know if you find any bugs or issues. It uses Python3 and so far I have tested it on RHEL8/RHEL9 clones and Ubuntu 24.04 on X86_64 and ARM64 platforms.
If you don’t need programmatic access, then you could just read the man pages or kernel header files, but there are a couple of handy “system call reference” websites out there too:
That’s all for now. Make sure that you go and star my 0x.tools GitHub repo - the more stars I get, the sooner I’ll release the next set of tools!
