mateusz@systems ~/book/page-cache $ cat section.md

The Linux Page Cache

The page cache is fundamental to understanding Linux filesystem performance. It's a unified cache in RAM that stores recently accessed file data, eliminating expensive disk IO for repeated reads and buffering writes for efficient batching.

# Architecture and Role

For buffered file IO, the kernel uses the page cache between the VFS and filesystem implementation: cache hits can be served from RAM, while misses initiate filesystem IO. Direct-IO paths such as O_DIRECT can bypass the page cache.

Key kernel structure: struct address_space (defined in include/linux/fs.h). This represents the pagecache for a particular inode. It maintains a radix tree (now xarray in newer kernels) mapping file offsets to cached pages.

# Read-Ahead Mechanisms

When the kernel detects sequential reads, it speculatively reads additional pages beyond what was requested. This read-ahead reduces latency for streaming workloads by keeping the IO pipeline full.

The read-ahead window typically grows as sequential access continues. When requests are treated as random, the kernel avoids expanding them into speculative read-ahead, limiting wasted memory and IO bandwidth.

Tunables: /sys/block/<device>/queue/read_ahead_kb controls maximum read-ahead size.

# Dirty Page Writeback

Writes to files are initially buffered in the page cache as "dirty" pages. The kernel flushes dirty pages to disk asynchronously, batching writes for efficiency. This improves write throughput but introduces complexity around durability and consistency.

Key sysctls controlling writeback behavior:

  • vm.dirty_ratio: Percentage of available memory at which a writing process starts writeback and may be throttled
  • vm.dirty_background_ratio: Percentage of available memory at which background writeback starts
  • vm.dirty_writeback_centisecs: Interval, in centiseconds, between periodic flusher wakeups
  • vm.dirty_expire_centisecs: Age after which dirty data becomes eligible for periodic writeback

Code path: mm/page-writeback.c implements the writeback logic. The wb_writeback() function is the core writeback worker.

# Page Cache Pressure and Reclaim

The page cache can grow into otherwise available RAM. Clean cache folios can be discarded; dirty filesystem folios must be cleaned by writeback before eviction. When free-memory watermarks fall, a per-node kswapd performs background reclaim, while allocation paths can also enter direct reclaim. Page selection uses the classic active/inactive LRU approximation or, when configured, multi-generational LRU (MGLRU). Under memory pressure, page reclaim can become a bottleneck.

# Filesystem-Specific Caches

Filesystems relate to the page cache in one of two ways. Most use it directly: file data lives in kernel-managed pages, and the filesystem plugs into the standard read and write paths. Others bring their own cache, managing a dedicated region of memory—often pinned, or living in a userspace process—instead of, or alongside, the page cache. The two local examples below show the second pattern; the network filesystems in the next section show how the same choice plays out across a cluster.

  • ZFS ARC (Adaptive Replacement Cache): ZFS manages its own cache with more sophisticated replacement policies than simple LRU. Balances between recently used and frequently used data. Can cause confusion when monitoring memory usage—the ARC appears as used memory but is reclaimable.
  • XFS buffer cache: XFS uses the page cache for file data but maintains separate metadata buffers for structural information (inodes, directories, extent maps).

# Caching Across Network Filesystems

A network filesystem caches data on the client to avoid paying network latency on every access. Two questions capture most of what matters for performance and correctness: where does that cached data live, and how is it kept consistent when several clients touch the same file?

On their ordinary buffered paths, NFS, Lustre, default-mode WekaFS, and VAST use the Linux page cache; ordinary GPFS IO uses its pinned pagepool, and OpenZFS uses ARC. mmap can take a separate path: OpenZFS mapped files use both the page cache and ARC, while GPFS handles mmap separately from its normal pagepool path. The systems differ most under concurrent sharing, where each protocol determines when a cached copy becomes stale.

FS Cached data Metadata cache Page cache?
ext4/XFS page cache inode/dentry Inside
ZFS ARC + L2ARC ARC metadata Outside for ordinary IO; mmap also uses page cache
NFS page cache attr cache (timed) Inside
Lustre page cache MDS lock cache Inside [1]
GPFS pagepool pagepool / token Outside for ordinary IO [2]
WekaFS page cache dentry (timed) Inside
VAST page cache NFS attr cache Inside [3]

[1] Pages are cached only while the client holds the matching LDLM extent lock.
[2] Ordinary GPFS IO uses its own pinned pagepool; mmap follows a separate path.
[3] The client uses the NFS page cache; the CNode write buffer (SCM) lives server-side.

FS Coherence model Read-only / shared read Concurrent read/write fsync / durability
ext4/XFS single-node POSIX coherent (one page cache) coherent through one kernel/page cache; application synchronization still required fsync to journal/disk
ZFS single-node POSIX + CoW coherent (ARC) coherent through one kernel; CoW does not make arbitrary concurrent updates atomic ZIL (sync); TXG batch (async)
NFS close-to-open cache; revalidate on open may remain stale until revalidation [4] WRITE plus COMMIT when needed on fsync/writable close
Lustre strict POSIX (LDLM) shared PR extent locks blocking callback → flush/inval flush to OST on revoke/fsync
GPFS strict POSIX (tokens) shared-read tokens token revoke → flush/inval [5] write-behind from pagepool
WekaFS strong POSIX (coherent) cache freely invalidate on remote access [6] write-back; fsync flushes [6]
VAST close-to-open (NFS) same as NFS NFS close-to-open; stricter opt. CNode buffer → SCM (durable)

[4] Ordinary close-to-open caching does not continuously invalidate another client's cache; NFSv4 delegations add server callbacks for exclusive caching. See the Close-to-Open deep dive for details.
[5] Tokens are byte-range granular, so non-overlapping writes can proceed in parallel.
[6] WekaFS default "write cache" mode is write-back; "read cache" mode is write-through and bypasses the page cache once a file is write-shared.

NFS caches file data and attributes per client and implements close-to-open consistency. open() revalidates against the server when required (NFSv3 commonly uses GETATTR or ACCESS), invalidating cached pages if the file changed; closing a writable descriptor drives dirty-page writeback. Between revalidation points a client can observe stale data. NFSv4 delegations let the server hand a client the right to cache aggressively and recall it with a callback when another client conflicts. See the close-to-open deep dive for the full protocol.

Lustre also uses the client's page cache, but a client may keep a byte range cached only while it holds the matching LDLM extent lock—a read (PR) lock for shared reading, a write (PW) lock for writing. When another client needs a conflicting lock, the server sends a blocking callback; the holder flushes any dirty pages and drops the cached range before releasing the lock. The result is strict POSIX coherency, paid for in lock traffic when clients write-share the same objects.

GPFS (IBM Storage Scale) keeps ordinary file IO outside the Linux page cache. Each node keeps data and metadata in the pagepool, a dedicated block of pinned memory it manages itself, and coordinates access with a distributed byte-range token manager; mmap follows a separate path. A node caches a region only while it holds the token; a conflicting access revokes the token and forces a flush or invalidation. Because tokens are byte-range granular, non-overlapping writes to the same file can proceed in parallel.

WekaFS caches file data in the Linux page cache, while a userspace client process (built on DPDK) handles the network path to the backends. Its default write cache mode is write-back: a write is acknowledged as soon as it lands in the page cache and is flushed to durable storage in the background, and coherency is preserved by invalidating a client's cached pages as soon as another client accesses the same file. An alternative read cache mode caches only reads (write-through) and steps aside—bypassing the page cache—once a file is write-shared. Metadata freshness is bounded by a timer (dentry_max_age_positive, one second by default) rather than by a lock.

VAST is reached through a standard NFS client, so on the client side it behaves like NFS: file data sits in the page cache under close-to-open consistency (stricter modes are available). Its distinctive cache is on the server—the stateless CNodes buffer incoming writes in Storage Class Memory on the DBoxes (durable, but a potential bottleneck under sustained write load) before destaging to QLC flash.

For contrast, a local filesystem has none of this to coordinate. ext4 and XFS keep file data in the page cache (XFS holds metadata in separate buffers); OpenZFS uses ARC for ordinary IO and also uses the page cache for mmap. A single kernel mediates all access, so concurrent readers and writers are coherent for free—the cache is the one authoritative copy.

Across these systems one pattern recurs: a client may cache data only while it holds some right to do so, and must flush or discard the cache when that right is revoked or expires. The right takes different names—an NFSv4 delegation recalled by a server callback, a Lustre LDLM lock revoked by a blocking callback, a GPFS token revoked by the token manager, a WekaFS cache entry invalidated on conflicting access—but the mechanism is the same. NFS without delegations is the outlier: it has no revocation channel, so it falls back to revalidating on open() and bounding staleness with a timer. That timer-based approach also bounds metadata staleness in NFS (acregmin/acregmax) and WekaFS (dentry_max_age_positive), whereas Lustre and GPFS tie metadata validity to the same lock or token layer as the data.

Durability follows the same split, and the recurring caveat is that a successful write() only means the data reached a cache—fsync() is what ties it to durable storage. NFS may send UNSTABLE WRITE RPCs and issues COMMIT when required; fsync() and writable close() drive synchronous writeback and commit. Lustre flushes a client's dirty pages to the OSTs on lock revocation or fsync(); GPFS destages from the pagepool; WekaFS's write-back mode flushes on fsync() (its read-cache mode is write-through); and VAST acknowledges once data is in SCM.

These caching and coherence choices are the mechanism behind the consistency and failure behavior compared in Distributed and Network Filesystems.

One access path deliberately left out of this comparison: memory-mapped IO interacts with each of these client caches on its own terms—coherently on some filesystems, not at all on others. See mmap coherence across filesystems.

# Observing Page Cache Behavior

The /proc/meminfo file exposes page cache statistics:

  • Cached: In-memory cache for files, including tmpfs, excluding SwapCached (not clean-only)
  • Dirty: Modified pages not yet written to disk
  • Writeback: Pages currently being written to disk
  • Mapped: Pages mapped into process address spaces (via mmap)

Tools like vmtouch can show which specific files are cached and even force files into or out of the cache for testing.