mateusz@systems ~/book/filesystem-hacks $ cat section.md

Filesystem Hacks

Unconventional filesystem techniques that solve real performance and deployment problems. These aren't theoretical edge cases—they're battle-tested approaches used in production systems.

squashfs: Reducing I/O Bottlenecks

squashfs is a compressed read-only filesystem. While commonly used for LiveCDs and snap packages, its most interesting applications exploit its ability to transform thousands of small file operations into efficient block reads.

Technique 1: squashfs on NFS. Compiler Explorer faced a challenging performance problem: serving more than 2,000 SquashFS images containing compilers, tools, and libraries from NFS. Each compilation triggered thousands of file accesses (headers, libraries, binaries), and even with aggressive caching, NFS metadata operations created severe latency. Their solution: pack each compiler into a squashfs image, store the images on NFS, and mount them via loop devices. This replaces many per-file NFS metadata and data operations with block reads against each image; SquashFS metadata, decompression, and filesystem caching are then handled locally, while the image itself still resides on NFS. Reference: CEFS blog post

Traditional NFS (slow) Client NFS stat header.h open header.h read header.h ... thousands more ... squashfs on NFS (fast) Client NFS read SquashFS blocks (decompress locally) headers cached in RAM minimal network round trips
Traditional NFS exposes per-file metadata and data access to network-filesystem latency. Packaging a toolchain as a SquashFS image turns those operations into reads of image blocks, with SquashFS metadata, decompression, and caching handled locally; it does not reduce a build to one network read.

Technique 2: squashfs in RAM for Python imports. Python's import system stats dozens of potential locations for each module. For large environments (anaconda, deep learning frameworks), this creates a "stat storm" that hammers filesystems. Packing the entire environment into squashfs and mounting it on tmpfs (in RAM) eliminates this overhead—all file metadata is instantly available from memory.

tmpfs: Isolating I/O from Performance Testing

tmpfs is a memory-backed filesystem (typically mounted at /dev/shm). Data lives in RAM and swap, making it extremely fast but volatile. Beyond the obvious use for /tmp, tmpfs is invaluable for performance testing and fast build directories.

Performance testing workflow: When benchmarking code that does file I/O, use tmpfs to eliminate disk as a variable. If your test is still slow on tmpfs, you know the bottleneck isn't I/O—it's your code. Example: dd if=/dev/zero of=/dev/shm/test bs=1M count=1024 writes at memory speed, perfect for testing serialization logic without disk interference.

Build acceleration: Compilation generates millions of intermediate files (.o objects, .d dependency files). Keeping build directories on tmpfs can dramatically speed up builds, especially on systems with slow disks or network-mounted home directories.

⚠ ramfs vs tmpfs

There's also ramfs, which is similar to tmpfs but doesn't respect size limits and never swaps. It's slightly faster (no size accounting overhead) but dangerous—accidentally writing too much will OOM-kill your system. Stick with tmpfs unless you absolutely need guaranteed RAM speed and can control size precisely. ramfs is a sharp tool that can cut you.

Loop Devices: Files as Block Devices

Loop devices let you mount a file as if it were a block device. This is fundamental for working with disk images, testing filesystems, and modern package formats.

Common uses: Testing filesystem behavior without dedicating real partitions (dd if=/dev/zero of=test.img bs=1M count=1024 && mkfs.ext4 test.img && mount -o loop test.img /mnt). Creating VM disk images. Mounting ISO images without burning them to physical media. Ubuntu's snap packages are squashfs images mounted as loop devices—when you install a snap, you're creating a loop mount.

disk.img (just a file) /dev/loopN (block device) mount -o loop /mnt/test (filesystem)
A regular file (disk.img) mounted with mount -o loop is attached to a free loop device (for example, /dev/loop0), represented here as /dev/loopN, which can then be mounted like any other block device at /mnt/test.

Encrypted filesystems: Loop devices combined with dm-crypt allow file-based encryption. The loop device presents the encrypted file as a block device, dm-crypt decrypts it, and you mount the result—all transparent to applications.

Bind Mounts: Same Directory, Multiple Locations

Bind mounts (mount --bind /source /target) make the same directory tree appear in multiple locations. Unlike symlinks, bind mounts work at the VFS layer and present normal files rather than symbolic links during pathname access, although applications can identify the mount through interfaces such as /proc/self/mountinfo.

Chroot and containers: Container runtimes use mount namespaces plus several mount types. Bind mounts specifically expose selected host paths and user-specified volumes; Docker and systemd-nspawn both support such host-path bind mounts.

NixOS sandboxing: Nix builds packages in isolated environments. The Nix store (/nix/store) is bind-mounted into build sandboxes along with specific dependencies, ensuring builds can't access undeclared dependencies. nix-user-chroot uses bind mounts to run Nix without root privileges.

Before /data/ file1.txt file2.txt /chroot/ (empty) After: mount --bind /data /chroot/data Both paths point to same files: /data/file1.txt /chroot/data/file1.txt (same inode)
mount --bind /data /chroot/data makes /chroot/data/file1.txt and /data/file1.txt refer to the same inode—the same file, reachable from two paths and presented as a normal file during pathname access, although mount topology remains observable.

Sparse Files: Huge Files, Small Space

Sparse files contain "holes"—ranges of zeros that don't consume disk space. When you read a hole, the filesystem returns zeros; when you write to a hole, it allocates real blocks. This is fundamental for virtual machine disk images and database files.

Creating sparse files: dd if=/dev/zero of=sparse.img bs=1 count=0 seek=10G creates a 10 GiB file (on a filesystem that supports sparse files) that uses almost no disk space. As the VM writes data, blocks are allocated on demand.

Logical view data hole (zeros) data 0 GiB 100 MiB 10 GiB Physical reality data data (about 100 MiB used) (about 100 MiB used) ls -lh: 10 GiB du -h: about 200 MiB
A sparse file's logical view spans 10 GiB with data at the start and end and a zero-filled hole in between; in this simplified example, approximately 200 MiB of data blocks are allocated. ls -lh reports logical size, while du -h reports allocated space and may include allocation-granularity and metadata overhead.

Gotchas: When sparse files materialize. Sparse files can accidentally expand to full size:

  • GNU cp defaults to --sparse=auto and normally attempts to preserve holes; --sparse=always forces sufficiently long zero runs to be represented as holes where supported
  • dd if=sparse.img of=copy.img without conv=sparse does the same
  • Copying a sparse file to a raw block device normally writes zero blocks for the source holes; it does not change the device's already provisioned physical capacity. Skipping those writes is safe only when the destination regions are known to contain zeros
  • Some backup tools don't preserve sparseness by default (check rsync --sparse, tar -S)

FUSE: Filesystems in Userspace

FUSE (Filesystem in Userspace) lets you implement filesystems as regular programs instead of kernel modules. The kernel FUSE module passes filesystem operations (open, read, write) to a userspace daemon, which handles them however it wants.

Application Kernel VFS FUSE kernel module Userspace daemon Actual storage
FUSE routes filesystem calls from the kernel VFS into a userspace daemon, which implements the filesystem logic and talks to whatever actual storage backs it—letting filesystems be ordinary programs instead of kernel modules.

Real-world examples:

  • sshfs: Mount remote directories over SSH. SSHFS uses the server's SFTP subsystem, which is enabled on most SSH servers but can be disabled or restricted. Useful for development against remote machines.
  • rclone mount: Mount cloud storage (S3, Google Drive, Dropbox) as a local filesystem. Reads and writes go to the cloud transparently.
  • encfs: An encrypted FUSE filesystem whose backing directory stores encrypted files and whose mount exposes plaintext. As of February 2026, the maintainer describes the Rust rewrite as alpha, says the old C++ implementation is unmaintained, and recommends newer alternatives for new encrypted filesystems.
  • Custom filesystems: Expose databases as directories (each row is a file), implement virtual filesystems for specific applications, or create read-only views of complex data structures.

Trade-off: FUSE lets filesystem daemons use ordinary userspace development and debugging tools, with bindings available for multiple languages. A daemon failure normally breaks its FUSE connection rather than running failed filesystem code in the kernel, while kernel/userspace request handoffs and data copies can add overhead. Whether that trade-off is acceptable depends on the workload.