ZFS deserves special attention as a local filesystem that combines traditional filesystem responsibilities with volume management, offering features rarely found together: end-to-end checksumming, integrated RAID, atomic snapshots, and transparent compression. Originally developed by Sun Microsystems for Solaris, OpenZFS now runs on Linux, BSD, and other platforms.

# Key Features

• End-to-end checksumming: Every block has a checksum. Silent data corruption (bit rot) is detected and, with redundancy, automatically repaired. Other checksumming filesystems, including Btrfs, provide similar detection and repair when redundant copies are available.
• Copy-on-write (CoW) architecture: Writes never overwrite existing data. New data goes to new blocks, metadata updated atomically. Enables fast, atomic snapshots and avoids overwriting live filesystem blocks in place.
• Integrated volume management: No need for LVM or mdadm. Create storage pools from multiple devices, dynamically allocate datasets (similar to filesystems). Devices can be added online; removal is supported only for eligible vdev types and pool layouts.
• RAID-Z: RAID-Z1, RAID-Z2, and RAID-Z3 provide single-, double-, and triple-parity distributed RAID. RAID-Z is broadly similar to RAID-5/6, avoids their write hole, and uses variable-width stripes rather than traditional partial-stripe updates.
• Snapshots and clones: Instant, space-efficient snapshots that work like "git for data." Take snapshots trivially, send only deltas between snapshots across systems, and maintain full version history of your filesystem. Writable clones enable testing or rollback scenarios. ZFS's incremental send/receive efficiently transmits only changed blocks, making it a powerful tool for managing and replicating filesets.

  $ zfs send -i @tuesday @wednesday | \
      ssh server-b zfs receive backup/data

Here @tuesday is the base snapshot (what server-b already has) and @wednesday is the target snapshot being sent. Only the delta (30GB of changes) is transmitted, not the full dataset—snapshots share unchanged blocks via copy-on-write, so replication stays space-efficient on both ends.

The efficiency of ZFS snapshots comes from its copy-on-write architecture. When you create a snapshot, no data is actually copied—instead, the snapshot simply references the existing data blocks. As new writes occur, only the modified blocks are written to new locations, while both the current filesystem and snapshots continue referencing the unchanged blocks. This means snapshots are nearly instantaneous and consume minimal space initially, growing only as the filesystem diverges from the snapshot state.

• Result: @snap1 preserves the original state (Blocks 1, 2, 3); the current filesystem uses Blocks 1, 4, 3; only Block 4 is new storage since Blocks 1 and 3 are shared; space used is the original data plus one modified block.

• ARC (Adaptive Replacement Cache) and L2ARC: ZFS uses a sophisticated multi-tier caching system that dramatically improves read and write performance while maintaining filesystem guarantees. The ARC is ZFS's primary RAM-based cache, more intelligent than simple LRU (Least Recently Used) caching. It balances two lists: recently used data (MRU) and frequently used data (MFU), adapting to workload patterns. This means a large file scanned once won't evict frequently-accessed small files from cache. For writes, ZFS buffers them in RAM and coalesces them into transaction groups, allowing for efficient sequential writes even when applications issue random writes. Synchronous operations are acknowledged after the required ZFS Intent Log (ZIL) records reach stable storage; their normal transaction-group state is committed later. L2ARC (Level 2 ARC) writes eligible ARC buffers to optional cache devices, creating a secondary cache tier between RAM and the main pool. Its benefit depends on the workload, and its index consumes some RAM; persistent L2ARC can rebuild its index after pool import.
• Compression and deduplication: ZFS supports transparent compression, including LZ4 and Zstandard; compression trades CPU work for lower storage and IO and should be measured on the target workload. Deduplication is available, but its dedup table can require substantial memory and storage, so size and test it before enabling it.

• ARC Algorithm: MRU (Most Recently Used) tracks data accessed once recently; MFU (Most Frequently Used) tracks data accessed multiple times. Scan-resistant—large sequential scans don't evict hot data. Adapts the split between MRU and MFU based on workload. The ARC maximum is tunable and its default depends on platform and OpenZFS release; with zfs_arc_max=0, OpenZFS 2.3.0 and later on Linux and FreeBSD use the larger of system memory minus 1 GiB and five-eighths of system memory.
• L2ARC (Level 2 ARC): Optional secondary cache tier fed with eligible ARC buffers and indexed by in-memory metadata. With persistent L2ARC, the index can be rebuilt after pool import.
• ZIL (ZFS Intent Log): Handles synchronous operations (O_SYNC, fsync, etc.); acknowledgment follows the required stable log commit, while normal transaction-group state is committed later. Can use a dedicated fast device (SLOG) for performance.

# Trade-offs and Considerations

• RAM requirements: Rule of thumb: 1GB RAM per TB of storage for basic usage, more for deduplication. ARC can be tuned but requires careful consideration. Low-RAM systems may struggle.
• Performance characteristics: Excellent for sequential I/O and large files. Random writes can fragment due to CoW. Throughput sensitive to recordsize setting (default 128KB, tunable per-dataset).
• Licensing: OpenZFS is CDDL-licensed, and the project documents CDDL/GPLv2 incompatibility as preventing distribution inside a combined Linux-kernel binary. Linux distributions therefore package OpenZFS separately; Ubuntu supplies the zfsutils-linux package in its repositories.
• Maturity on Linux: OpenZFS on Linux is production-ready and widely deployed, but originally designed for Solaris. Some features (e.g., certain performance tunings) work differently on Linux.
• Use as Lustre backend: Can provide OST storage with checksumming benefits. However, XFS typically offers better raw performance. Choose ZFS for data integrity, XFS for maximum throughput.

# Understanding Recordsize

One of the most misunderstood aspects of ZFS is recordsize and how it differs from block size. The recordsize (default 128KB) is the maximum unit of data ZFS will write at once, but crucially, ZFS adapts recordsize to file size. A 4KB file is written as a 4KB record, not 128KB—there's no internal fragmentation or waste. This adaptive behavior surprises many people who expect fixed-size allocation.

With the dataset's recordsize left at the default 128KB, actual allocation adapts to each file's size:

Key insight: Small files don't waste space, even with large recordsize.

When recordsize matters: Recordsize primarily impacts I/O efficiency, not space usage. There are two key workload patterns to consider:

Workload Pattern 1: Whole-File Access
    Application reads/writes entire files at once
    Examples: Media files, archives, backups, ML datasets

    Large recordsize (e.g., 1MB) benefits:
        ✓ Better compression ratio (more data per compression unit)
        ✓ Fewer metadata operations (one record vs many)
        ✓ More efficient sequential I/O
        ✓ Small files still allocate correctly (adaptive)

    Recommendation: Use large recordsize (512KB - 1MB)
        $ zfs set recordsize=1M tank/media

Workload Pattern 2: Random/Partial Access
    Application reads/writes from middle of files
    Examples: Databases, random-access data structures, VMs

    Problem with large recordsize:
        Read 4KB at offset 50KB from file:
            - ZFS must read entire 128KB record from disk
            - Then return requested 4KB to application
            - Wasted I/O if accessing random locations

        Write 4KB at offset 50KB:
            - ZFS must read 128KB record (read-modify-write)
            - Modify 4KB portion
            - Write entire 128KB back (copy-on-write)
            - Extra I/O amplification

    Aligned recordsize = block size benefits:
        Read/write operations match underlying device block size
        Avoid read-modify-write cycles
        Predictable I/O patterns

    Recommendation: Match recordsize to access pattern
        Databases with 8KB pages:  $ zfs set recordsize=8K tank/db
        VMs with 4KB blocks:       $ zfs set recordsize=4K tank/vms
        Default 128KB is good for general mixed workloads

Compression interaction: Larger recordsize improves compression ratios because compression algorithms work on entire records. A 1MB recordsize with highly compressible data can achieve better ratios than 128KB, even for the same data. However, the first access to any part of a compressed record requires decompressing the entire record, so random access patterns may see worse performance with large recordsize, even accounting for compression.

# When to Choose ZFS

• Data integrity critical: Archive systems, long-term storage, media libraries where bit rot detection matters. End-to-end checksums detect corruption, and redundant pools can repair a damaged copy automatically.
• Snapshot workflows: Backup targets benefiting from incremental send/receive. Development environments needing instant rollback. VM/container storage with snapshot-based provisioning.
• Integrated RAID: Avoiding LVM/mdadm complexity. Home servers or NAS appliances where ZFS's all-in-one approach simplifies management.
• Avoid when: System has very limited RAM (<4GB for typical workloads). Need maximum random write performance (databases may prefer XFS/ext4). Kernel-space requirements preclude third-party modules.