Root-causing mount(2) with eBPF

Anthony A. Vardaro, Jul 2026

I have found that eBPF is the fastest way for me to root-cause a busted mount(2) invocation. The same method works for its modern variant, fsconfig(2) with FSCONFIG_CMD_CREATE.1

The mount(2) syscall fails in weird ways. The kernel runs a fairly rigorous sequence of permission, kernel namespace, and compatibility checks before it hands superblock creation off to the underlying file system. Granular use of kernel namespaces, particularly user_namespaces(7), mount_namespaces(7), and network_namespaces(7), increases the surface area for failure and makes root-causing less tractable. A substantial portion of the mount process occurs below the kernel's VFS layer, in the underlying file system. Some file systems do network I/O during the mount process (nfs4, lustre), and they inherit all the failure modes that come with it. Others interact weirdly with kernel namespaces late in the mount process (overlayfs, also nfs4). In containerized compute environments, mount(2) sits in the critical path of launching a workload, so fast root-causing of this class of failure is fairly important.

A colleague at Netflix introduced me to the idea of using eBPF to trace the return values of interior kernel functions. Up until that point, my method was to trace the userspace-visible errno back through the kernel code by hand, which was both slow and lossy. That works for errnos with clear origins (ENODEV is fairly clear). It's ineffective for overloaded errnos like EINVAL. The mount path in fs/namespace.c alone has close to a hundred places that return EINVAL, super painful.2 With eBPF, we can attach kretprobes to the important VFS functions (sget_fc(), vfs_get_tree(), etc.) and to the relevant file system creation functions (do_nfs4_mount(), nfs4_create_server(), ovl_fill_super(), etc.). We can watch the syscall boundary with the sys_exit_* tracepoints. Together, these show where inside the kernel an errno was born.

A Simplified Trace Script

Here's an example of what I mean. At syscall entry, we stash the mount(2) arguments. In flight, kretprobes record failing return values from "interesting" kernel functions. At exit, we print the arguments and the deepest failure. Something to note, kretprobes fire in return order, and the deepest function returns first, so the first failure we print is the deepest one.

The probes are checkpoints in the mount(2) pipeline, which for a new mount looks like this. path_mount() runs the LSM hook (security_sb_mount()) and checks that the caller is capable in its mount namespace. do_new_mount() resolves the file system type and builds an fs_context. vfs_get_tree() creates the superblock, sget_fc() finds-or-allocates it, and the file system fills it (this is where something like ovl_fill_super() in overlayfs would live). Finally, graft_tree() attaches the finished mount into the namespace tree. There's more complexity available, but for my purposes I've found this is the appropriate level of depth relevant to productively debug this stuff, since these points constitute what I consider to be the common failure culprits. 

#!/usr/bin/env bpftrace

tracepoint:syscalls:sys_enter_mount
{
	@source[tid] = str(args.dev_name);
	@target[tid] = str(args.dir_name);
	@fstype[tid] = str(args.type);
	@failed[tid] = probe;
	@err[tid] = (int64)0;
}

/*
 * Checkpoints on "interesting" VFS kernel functions.
 */
kretprobe:security_sb_mount,
kretprobe:graft_tree,
kretprobe:vfs_get_tree,
kretprobe:ovl_fill_super
{
	/*
	 * These return an int, and the upper 32 bits of the return
	 * register are garbage, so chop them off before comparing.
	 */
	if (@err[tid] == 0 && (int32)retval < 0) {
		@failed[tid] = probe;
		@err[tid] = (int64)(int32)retval;
	}
}

/* These return pointers. IS_ERR() means -4095 <= retval < 0. */
kretprobe:sget_fc,
kretprobe:fc_mount
{
	if (@err[tid] == 0 && (int64)retval < 0 && (int64)retval > -4096) {
		@failed[tid] = probe;
		@err[tid] = (int64)retval;
	}
}

tracepoint:syscalls:sys_exit_mount
{
	if (args.ret < 0) {
		printf("%-16s mount(%s, %s, %s) = %d\n", comm,
		    @source[tid], @target[tid], @fstype[tid], args.ret);
		if (@err[tid] != 0) {
			printf("%-16s deepest failure: %s = %d\n", "",
			    @failed[tid], @err[tid]);
		}
	}
	delete(@source[tid]);
	delete(@target[tid]);
	delete(@fstype[tid]);
	delete(@failed[tid]);
	delete(@err[tid]);
}

END
{
	clear(@source);
	clear(@target);
	clear(@fstype);
	clear(@failed);
	clear(@err);
}

This is a shortened version of why-did-mount-fail.bt, which is the tool I actually use, because it probes more things. I didn't author this tool, full credit goes to Tycho.

Note on the (int32) casts, bpftrace reads retval as a raw 64-bit register, so an int return of -22 reads as 4294967274 and never passes a signed comparison.3

The deepest failing probe is usually strong evidence, sometimes it is not a definitive root cause. In some error paths, errnos are rewritten or handled on the way back up the call chain, which is why I prefer dotting the kretprobes along the call stack as opposed to isolating the lowest function.

This is safe to leave attached to a busy production host. Although, I tend to run this ad-hoc rather than leaving it always-on. The probes only fire on mount activity, which is relatively rare, and a kprobe hit costs on the order of a microsecond, not enough to make a meaningful dent in workload launch latency.

Examples
Misplaced workdir (EINVAL)

As an example, let's pretend this host runs a container orchestration system with a hypothetical container runtime that supports our company's workloads. The runtime assembles each rootfs with overlayfs, ephemeral sandbox state on /run (a tmpfs) and durable state on /var/lib (backed by a real disk).

The runtime makes the equivalent of this mount(8) call: 

$ sudo mount -t overlay overlay \
    -o lowerdir=/var/lib/runtime/images/base/rootfs,upperdir=/run/runtime/sandboxes/42/upper,workdir=/var/lib/runtime/sandboxes/42/work \
    /run/runtime/sandboxes/42/rootfs
mount: /run/runtime/sandboxes/42/rootfs: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

$ sudo strace -e trace=mount mount -t overlay overlay \
    -o lowerdir=/var/lib/runtime/images/base/rootfs,upperdir=/run/runtime/sandboxes/42/upper,workdir=/var/lib/runtime/sandboxes/42/work \
    /run/runtime/sandboxes/42/rootfs
mount("overlay", "/run/runtime/sandboxes/42/rootfs", "overlay", 0, "lowerdir=/var/lib/runtime/images"...) = -1 EINVAL (Invalid argument)

Running strace gets you as far as EINVAL, but of what exactly? Super confusing.

With our script attached: 

$ sudo bpftrace trace-mount.bt
Attaching 9 probes...
mount            mount(overlay, /run/runtime/sandboxes/42/rootfs, overlay) = -22
                 deepest failure: kretprobe:ovl_fill_super = -22

Failing at this level suggests the mount survived the generic VFS checks and died inside overlayfs's superblock setup.

Knowing the failed function, we can grep the kernel code with more precision, and land somewhere around here: 

// fs/overlayfs/super.c, in ovl_get_workdir() (as of v7.2)
err = -EINVAL;
if (upperpath->mnt != workpath->mnt) {
	pr_err("workdir and upperdir must reside under the same mount\n");
	return err;
}

Root cause is bad input. upperdir landed on the /run tmpfs, and workdir stayed on ext4. But overlayfs needs both on the same mount, so that it can rename(2) between them. The fix is one line in the runtime's directory layout, aligning the workdir with the upper dir.

In fairness, overlayfs is polite enough to pr_err() that one into dmesg (and the code pointer I shared illustrates that). Not everything is, though. In the next example we can examine a failure that is less obvious.

Mount Leak (ENOSPC)

The fs.mount-max sysctl caps the number of mounts in a mount namespace. A container runtime or CSI driver that leaks mounts will eventually hit the cap on a long-running host. Upon hitting this cap, the sandbox mount fails with ENOSPC while df shows half the disk free. Kernel dmesg logs don't announce this, despite what mount(8) suggests: 

$ sudo mount -t tmpfs tmpfs /run/runtime/sandboxes/43/rootfs
mount: /run/runtime/sandboxes/43/rootfs: mount(2) system call failed: No space left on device.
       dmesg(1) may have more information after failed mount system call.

$ sudo bpftrace trace-mount.bt
Attaching 9 probes...
mount            mount(tmpfs, /run/runtime/sandboxes/43/rootfs, tmpfs) = -28
                 deepest failure: kretprobe:graft_tree = -28

The tmpfs portion built its superblock fine. The mount died later, while the VFS attached it to the mount namespace. A grep for ENOSPC in fs/namespace.c lands in count_mounts(), which enforces sysctl_mount_max. You can then use wc -l < /proc/self/mountinfo to verify the leak.

At first glance, this class of failure is deeply unclear (and could be considered misleading). The kernel reporting ENOSPC will point most operators to du/df, but using eBPF we can quickly isolate the real underlying failure with few distractions.

Footnotes
  1. The simplified script only traces mount(2). If your workload uses the new mount API, also watch the sys_exit_fsconfig and sys_exit_fsmount tracepoints. One note, mount(8) itself may use the new API depending on your util-linux version. LIBMOUNT_FORCE_MOUNT2=always (util-linux 2.40+) forces the classic syscall.
  2. Verified super rigorously with cat ./fs/namespace.c | grep "EINVAL" | wc -l, which returns 90 mentions on v6.17.
  3. People hit the same thing filtering on retval: bpftrace#668.
Email LinkedIn Blog About