Understanding how data flows from application to disk—and the role of buffering—is critical for performance optimization and correctness.

# Buffered IO (Default)

By default, buffered IO on regular files uses the page cache. Reads can be satisfied from cached folios; writes normally dirty cached folios and may return before storage writeback completes, although the write call can still block for allocation, faults, throttling, or other kernel work. O_SYNC and O_DSYNC add completion requirements.

Read path:

1. Application calls read()
2. VFS layer checks page cache for requested offset
3. Cache hit: Copy from cache to user buffer, return immediately
4. Cache miss: Read the required data and, for a sequential access pattern, potentially read ahead; populate the cache and copy to the user buffer

Code: mm/filemap.c:generic_file_read_iter()

Write path:

1. Application calls write()
2. VFS layer finds or allocates page cache pages for the write range
3. Copy data from user buffer to cache pages
4. Mark pages dirty
5. Return after the buffered write is accepted; this may precede storage writeback
6. Later: Writeback thread flushes dirty pages to disk

Code: mm/filemap.c:generic_perform_write()

Performance characteristics:

• Excellent for sequential workloads (read-ahead hides latency)
• Repeated access to same data is very fast (cache hits)
• Writes are batched and coalesced, improving throughput
• No special alignment requirements

# Direct IO (O_DIRECT)

O_DIRECT requests direct IO, minimizing page-cache effects and, when the filesystem supports the request, transferring data between user-space buffers and storage without the normal page-cache path. O_DIRECT alone neither bypasses hardware caches nor supplies O_SYNC durability.

Requirements:

• User buffer must be aligned to filesystem block size (usually 4KB)
• IO offset must be aligned to filesystem block size
• IO length must be a multiple of filesystem block size
• Violating alignment causes EINVAL errors

void *buf;
posix_memalign(&buf, 4096, 4096);  // Align to 4KB
int fd = open("datafile", O_DIRECT | O_RDONLY);
pread(fd, buf, 4096, 0);  // Read 4KB at offset 0

When to use Direct IO:

• Databases: Manage their own caching (e.g., InnoDB buffer pool, PostgreSQL shared buffers). Page cache would be redundant "double buffering."
• Large sequential reads: Streaming large files where data won't be reused. Avoids polluting page cache.
• Low-latency requirements: Eliminate cache management overhead for predictable latency.

Pitfalls:

• Small random reads/writes perform poorly (no buffering or merging)
• Application must discover and obey any per-file alignment constraints
• Direct IO does not use normal page-cache read-ahead
• Avoid overlapping direct and buffered IO on the same file: even when the filesystem maintains coherence, throughput can suffer

Common code paths include the legacy fs/direct-io.c implementation, the modern fs/iomap/direct-io.c implementation, and filesystem-specific direct-IO code.

# Synchronous IO Flags

Several flags control write durability:

• O_SYNC: Writes block until data and metadata are on stable storage. Expensive but ensures durability.
• O_DSYNC: Like O_SYNC but doesn't wait for metadata updates (e.g., file size, modification time) unless necessary for reading the data back.
• fsync(fd): System call to flush all dirty data and metadata for a file to disk. Blocks until complete.
• fdatasync(fd): Like fsync() but skips metadata updates when possible (similar to O_DSYNC).
• sync_file_range(): Starts and/or waits for page-cache writeback over a byte range according to its flags (flags 0 is a no-op); it does not flush metadata or device write caches and must not be used for data-integrity synchronization.

Performance implications: Synchronous writes serialize IO and force disk flushes, destroying write batching. Use sparingly—only when durability is critical (e.g., database transaction commits).

# Interaction with Filesystems

Different filesystems handle buffered vs direct IO differently:

• ext4: Supports buffered and direct IO; direct-IO eligibility and alignment are file- and feature-dependent, and supported kernels can report alignment with statx(STATX_DIOALIGN).
• XFS: Excellent direct IO support. Direct IO writes bypass page cache but may still update metadata.
• OpenZFS: OpenZFS 2.3 and later support O_DIRECT; with direct=standard, eligible direct requests bypass ARC (ZFS's Adaptive Replacement Cache). Fallback behavior for unaligned requests is version-dependent, with OpenZFS 2.4 providing a lightweight uncached-IO fallback.
• NFS: Direct IO still involves network round-trips. Can reduce client-side caching but doesn't eliminate all buffering.