On Linux, ordinary ELF executables and shared objects are file-mapped and demand-paged: the kernel ELF loader maps the executable's loadable segments MAP_PRIVATE, while the dynamic linker maps shared libraries. For conventional page-cache-backed buffered IO, read() copies data from page cache into a user buffer, whereas mmap() maps the cached file pages into the process. OpenZFS mmap and DAX are important exceptions to the literal one-copy model.

# The mmap() Call

mmap() maps a file (or a portion of one) into the process's address space:

int fd = open("datafile", O_RDWR);
char *data = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
data[42] = 'x';            /* modifies the file - no write() call */
munmap(data, length);

The PROT_* flags set page permissions (they must be compatible with how the file was opened); the last argument is the file offset, which must be a multiple of the page size. The interesting choice is the sharing mode:

• MAP_SHARED: your stores go to the file's own pages. They are visible to every other process mapping the file and are carried through to the file itself.
• MAP_PRIVATE: copy-on-write. The first store to a page gives you a private copy of that page; your changes never reach the file. The divergence is page-by-page—untouched pages keep following the file, and whether changes made to the file by others show through them is unspecified, so don't build on either behavior.

There is also MAP_ANONYMOUS, which maps zero-filled memory backed by no file at all—it is how allocators get large blocks from the kernel. That underlines the real nature of mmap(): it is the kernel's general-purpose "give me address space" primitive, and file mapping is one use of it.

# One Copy of Truth: mmap and the Page Cache

Without population or locking flags, mmap() primarily installs a VMA (virtual memory area) describing the file range and permissions; physical pages and page-table entries are then populated on faults. MAP_POPULATE and MAP_LOCKED can prefault pages. Mapping latency is therefore usually tied more to VMA bookkeeping than file length, but no fixed microsecond bound is guaranteed.

For conventional page-cache-backed filesystems, a resident MAP_SHARED page is a page-cache folio mapped directly into each process, so multiple processes share the same physical cache pages; a MAP_PRIVATE page diverges on its first write. This one-copy model is not universal: OpenZFS keeps page-cache and ARC copies synchronized for mmap, and DAX maps storage directly.

The diagram shows the two doors side by side. Through the read() door, two copies of the data exist—the cache page and your buffer—and they begin to drift apart the moment the file changes. Through the mmap() door there is only the cache page. The price of the second door is that you inherit the cache's lifecycle: pages arrive by fault, leave by reclaim, and are written back on the kernel's schedule—the subject of the next two sections.

# First Touch: the Page-Fault Path

Because mmap() allocated nothing, the first access to each mapped page traps into the kernel—the page-table entry is empty. The fault handler finds the VMA covering the faulting address, which names the file and offset, and looks the page up in the page cache. From there, two outcomes:

• Minor fault: serviced without IO—for example by installing a PTE for an already resident, up-to-date folio. A nominal cache hit can still wait on an in-flight folio.
• Major fault: requires IO, as when a cache miss initiates a file read. Readahead can make later first-touch faults minor. These categories do not imply fixed latency.

The kernel counts the two separately (minflt/majflt in /proc/<pid>/stat, ru_minflt/ru_majflt from getrusage()—see the observability section). A mapped scan that runs slow is usually a stream of major faults; the same scan warm is all minor faults.

Sequential faulting triggers the same readahead machinery as read(): the kernel detects the pattern and fetches ahead, converting would-be major faults into minor ones. You can steer it. madvise(MADV_SEQUENTIAL) and MADV_RANDOM adjust the readahead window, MADV_WILLNEED starts reads for a range now, and MAP_POPULATE at mmap() time asks the kernel to prefault the mapping up front—most of the first-touch cost paid at startup, though it is best-effort rather than a guarantee. These are the mmap-side analog of posix_fadvise() (see the IO modes section for the read-side paths).

# Writes: Dirty Pages and the Three Paths to Disk

A store instruction is not a system call, so how does the kernel even find out you wrote? By a trick: clean MAP_SHARED pages are mapped write-protected. Your first store to a page faults; the kernel calls the filesystem's page_mkwrite hook, marks the page dirty in the page cache, makes the page-table entry writable, and resumes. Later stores to that page run at full memory speed until the page is cleaned again—writeback and msync() re-write-protect what they flush, so the next store faults anew. Writes are therefore noticed page-by-page, each time a clean page is first dirtied—a write burst into a large mapping shows up as a burst of minor faults, and a long-lived mapping pays that burst again after every writeback cycle.

From that moment the page is dirty, and it is exactly like a page dirtied by write(). Three things can carry it to disk:

1. Explicit sync—msync(MS_SYNC) or fsync(fd): flush now, block until durable, report errors. A mapped page is a file page, so fsync covers it too. (MS_ASYNC has been a no-op since Linux 2.6.19—the kernel already tracks dirty pages and needs no hint.)
2. Background writeback: the same flusher machinery and dirty-timer tunables described in the page cache section. It will get there, silently, on its own schedule—and if it hits an IO error, nothing tells you at the time: the kernel records the error, and a later msync()/fsync() is what surfaces it.
3. Memory pressure: pressure can accelerate writeback indirectly, and a folio can be evicted after it becomes clean. Current reclaim does not itself write ordinary dirty filesystem folios.

Now the negative space, which is where the bugs live. On ordinary local filesystems, munmap(), close(), and process exit are not durability operations and do not synchronously flush merely because a mapping disappears. Network filesystems may add close-time behavior—for example, Linux NFS flushes on writable close. On a local filesystem the dirty pages survive in the page cache and background writeback eventually writes them—but "eventually" comes with no ordering, no timing, and no error report. For durability you need msync(MS_SYNC), or fsync(fd)—a mapped page is a file page, so fsync covers it. And MAP_PRIVATE pages are never written back at all, by definition.

# Coherence: Local vs Network Filesystems

On a local filesystem, mmap coherence is free. There is one page cache per file and one kernel mediating it, so a store through a mapping is immediately visible to read(), to write()-ers, and to every other process mapping the file—they are all touching the same physical page. What you do not get is atomicity or ordering: concurrent access to a mapped page has plain shared-memory semantics, and coordinating it is your job (see Synchronization Guarantees).

A network filesystem gives each client its own cache and integrates coherence at protocol-specific points, including open/close, page faults, writeback, invalidation, and server callbacks. An ordinary load or store is not itself a syscall, so live mapped-page coherence depends on whether the filesystem integrates its mapping and fault paths with that protocol.

For NFS, the answer is: only partly. The write side mostly works on Linux—stores through a mapping are noticed (the same write-protect trick from the previous section), and closing a descriptor opened for writing writes back all of the file's dirty pages, mapped ones included, per close-to-open. The gaps are at the edges: munmap() does not count as a close and flushes nothing, and—because the mapping outlives the descriptor—stores made after the last writable close() reach the server only whenever background writeback gets around to them, with no close-to-open ordering at all. msync(MS_SYNC), or fsync() on a still-open descriptor, remains the explicit sync point that does not depend on call order. Fresh open() and mmap() operations revalidate when required. Loads from an already resident live mapping do not themselves revalidate. After an NFS file lock is successfully acquired, current Linux clears stale cache state and revalidates mapped pages only when no read or write delegation is held. Absent such a coherency-triggering operation, mapped pages can remain stale indefinitely, and reopen/remap provides an explicit opportunity to refetch. NFSv4 delegation recalls coordinate conflicting access but do not make continuously shared mmap strictly POSIX-coherent.

Lustre taught its fault path about the protocol. Servicing a page fault acquires the same LDLM extent locks that protect read() and write(), and cached pages are flushed before a lock is released, so mapped access is coherent across clients. The price is that faults are page-granular and a single mapped IO may need more than one lock—random mmap workloads typically run slower than buffered IO and are a known tuning topic.

GPFS governs mapped access with the same distributed byte-range token protocol as everything else, so concurrent mmap writers coordinate at token granularity. But mmap is a long-standing special case in GPFS—normal IO is served from its own pinned pagepool rather than the Linux page cache, the mmap path has historically lagged in performance, and IBM has published a fix for incomplete flushes of concurrent mmap writes. Treat write-shared mmap on GPFS with suspicion.

WekaFS extends its POSIX semantics to mmap: the client invalidates cached pages as soon as another host touches the same data, so mapped files stay coherent across clients in both of its mount modes (the modes differ in write-back vs write-through durability, not in coherence). VAST is reached through the standard Linux NFS client—its optional high-performance client is a multipath extension of the kernel NFS driver, not a separate filesystem—so the NFS story above applies unchanged: no cross-client mmap coherence.

[1] OpenZFS double-caches mapped files: the page cache and the ARC each hold a copy that the filesystem keeps synchronized — correct, but it costs memory and coherence work.
[2] Fresh open() and mmap() revalidate when required, but mapped loads do not. After an NFS file lock is successfully acquired, current Linux clears stale cache state and revalidates mapped pages only when no read or write delegation is held; absent such a coherency point, live pages can remain stale. NFSv4 delegation recalls coordinate conflicting access but do not provide continuously shared strict mmap coherence. On the write side, closing a descriptor opened for writing writes mapped dirty pages back, but munmap() does not, and stores made after the last writable close are picked up only by background writeback.
[3] mmap is a documented trouble spot in GPFS: historically slower than regular IO, with an IBM-published fix for incomplete flushes of concurrent mmap writes.

This is the mmap column of the picture drawn in Caching Across Network Filesystems: the systems that hold a revocable right to cache (Lustre locks, GPFS tokens, WekaFS invalidation) extend that right to mapped pages; the systems that rely on NFS-style revalidation do not revalidate on ordinary mapped loads. A fresh open/mmap can revalidate when required. After an NFS file lock is successfully acquired, current Linux clears stale cache state and revalidates mapped pages only when no read or write delegation is held; a writable close() still flushes.

# Performance: When Mapping Beats Reading

Once a mapped page is resident and has a PTE, access is an ordinary memory instruction with no per-access syscall or copy. read() incurs a syscall and copies into a user buffer. mmap can also avoid a second long-lived application-cache copy; it roughly halves that component only when the read-based design would retain an equally sized user-space copy.

What mmap costs: first touches. A cold scan through a mapping faults page by page—1 GiB is 262,144 potential faults at 4 KiB—where a read() loop would issue a few large, readahead-friendly IOs. Page tables must be built and torn down: unmapping a large hot mapping triggers TLB shootdowns (inter-processor interrupts to every core that touched it), a real cost on many-core machines. TLB pressure itself grows with working-set size at 4 KiB per entry—on recent kernels the page cache can back file mappings with large page-cache folios, including XFS configurations whose filesystem geometry selects a larger minimum folio order, which helps, but support is filesystem-dependent and still evolving. And there is no asynchronous interface to a page fault: a major fault stalls the thread, full stop, where the read paths offer readahead, io_uring, and O_DIRECT for overlap and control. madvise() hints mitigate; they do not fix.

Rule of thumb: mmap wins for repeated, random access to hot data, for many processes sharing one big read-mostly structure, and whenever you want the page cache to be your application cache. read()/pread() wins for one-pass streaming, for latency-critical paths that cannot absorb a surprise major fault, and for anything that needs honest error handling—the Pitfalls section's subject.

# Use Cases

Executables and shared libraries

The kernel ELF loader maps executable load segments, and the dynamic linker maps shared libraries MAP_PRIVATE, file-backed and demand-paged. Code pages are read-only, so the copy-on-write never happens— every process mapping libc shares the same physical page-cache pages, and one copy of its code serves the whole machine. Demand paging is why a large binary starts fast: only demanded load-segment pages, plus pages fetched by readahead, need be read. cat /proc/self/maps shows the mappings (r-xp + a file path on every library). This is mmap on its home turf: read-only, hot, shared.

Databases

LMDB exposes the entire database through one long-lived map and returns reads by pointer straight out of it—no copies, no buffer pool. Note what it does not do: by default the map is read-only, and updates are written with pwrite()—protecting the file from stray pointers—and crash safety comes from copy-on-write pages plus two alternating meta pages ordered by fsync, never from assumptions about when mapped stores hit disk. SQLite's optional mmap mode (PRAGMA mmap_size) is similar: the map serves reads; modifications are copied to heap and written back through the normal write path—"mostly a benefit for queries," per its docs. The pattern that works: read through the map, write around it.

Shared-memory IPC

MAP_SHARED is the substrate for shared memory between processes: shm_open() + ftruncate() + mmap() in two processes yields a region both can store into, backed by tmpfs—page-cache pages with no durable file behind them. Map a regular file instead and the segment persists. Ring buffers, metrics segments, and lock-free queues are built this way. The stores are visible to the peer at cache-coherency speed; making them safe (atomics, memory ordering, futexes) is entirely your problem—see Synchronization Guarantees.

Large read-only datasets

Reference data, search indices, model weights: huge files that many processes—or many nodes—read but never write. Read-only mapping sidesteps most of this section's hazards (nothing is ever dirty, so the flush question disappears), and on Lustre or GPFS shared read locks and tokens let every node cache freely. Two costs to plan for at scale: the first-touch fault storm when a thousand ranks cold-start against the same file—prefault with MAP_POPULATE or MADV_WILLNEED, or stage to node-local NVMe or tmpfs first—and page-granular faulting, which amplifies random-access latency over a network compared to streaming a copy. On NFS-backed mounts (VAST included), read-only mmap of static files is fine; the coherence trap only bites when files change while mapped.

# Pitfalls

Most mmap bugs are one of these:

• Truncation means SIGBUS: if another process truncates the file, touching a mapped page beyond the new end of file raises SIGBUS. Mapping a file that someone else can shrink is a crash waiting for a schedule.
• IO errors mean SIGBUS too: read() hands you EIO to handle; a failed page fault delivers a signal to whatever instruction happened to touch the page. On unreliable media or network mounts, this alone is a reason to prefer read().
• The zero-filled tail: a file's last page is padded with zeros in memory when the size isn't page-aligned. Stores into that padding succeed—and are never written to the file. And touching pages wholly beyond EOF raises SIGBUS: you cannot grow a file by storing past its end. ftruncate() first, then map (or remap).
• The mapping outlives the fd: close(fd) after mmap() is legal and common—the mapping keeps working. The trap is believing the descriptor's lifecycle controls the data's: locally close() flushes nothing, and on NFS it writes back only what you stored before it—stores made through the surviving mapping afterward are flushed by nothing but background writeback.
• fork() shares your mappings: the child inherits every MAP_SHARED region live—parent and child see each other's stores from that moment on, an easy accidental IPC channel (and corruption channel) if the child was supposed to be independent.