Network filesystems allow multiple machines to share access to data, enabling collaboration and resource pooling. Understanding their trade-offs is critical: choosing the wrong filesystem for your workload can mean the difference between millisecond latencies and minute-long hangs, between graceful degradation and catastrophic data loss. This section introduces a framework for evaluating network filesystems across three key dimensions: semantics (what guarantees does the FS provide?), architecture (how does it scale and perform?), and failure modes (what breaks and how?). By understanding these dimensions, you'll know what to pay attention to when working with any network filesystem—even ones not covered here.

# Architecture Overview

Before diving into comparisons, let's visualize how these systems are structured. Architecture fundamentally determines scalability, failure modes, and performance characteristics.

NFS - Single Server Architecture

Performance: All operations funnel through a single server, limiting throughput to one machine's capabilities. Metadata operations (stat, readdir) and data I/O compete for the same resources, creating contention under mixed workloads.

Resiliency: Server failure makes the entire filesystem unavailable—no failover, no redundancy. Common in simple deployments where simplicity outweighs HA requirements.

Lustre - Separated Metadata and Data

Performance: Separating metadata and data enables massive parallel throughput—clients can hit multiple OSTs simultaneously for striped files, achieving aggregate bandwidth that scales with OST count. Metadata server (MDS) handles namespace operations independently.

Resiliency: MDS remains a single point of failure (unless using HA config), but OST failures only affect files stored on that target. Ideal for HPC workloads with large sequential I/O patterns.

GPFS - Shared Disk with Distributed Locking

Performance: Token-based locking enables cache coherency without constant server round-trips—nodes can cache data as long as they hold the appropriate token. Direct storage access eliminates data forwarding overhead.

Resiliency: No single point of failure—any NSD server can fail and others take over. Token manager is distributed across the cluster. Shared storage itself needs redundancy (RAID, replication). Trade-off: lock contention can hurt metadata-intensive workloads.

WekaFS - Distributed with Coherent Cache

Performance: Every cached-mode client can use its host's Linux page cache in RAM, minimizing backend reads for hot data. Cache coherency protocol ensures consistency without sacrificing speed—reads hit local cache, writes invalidate remote copies. Distributed backend provides parallel access paths.

Resiliency: Fully distributed architecture with no single point of failure. Data is protected by WEKA's configurable distributed RAID (for example N+2, N+3, or N+4), and failures trigger automatic rebuild. Optimized for modern cloud-native and AI/ML workloads requiring both high throughput and low latency.

VAST - Disaggregated Shared Everything (DASE)

Performance: VAST's "Disaggregated Shared Everything" (DASE) architecture separates stateless compute nodes (CNodes) from storage enclosures (DBoxes). Any CNode can access any DBox over NVMe-oF fabric, enabling global data reduction and eliminating isolated data islands. Supports standard NFS access plus optional nconnect, VAST-client multipath, and NFS over RDMA—simpler than parallel FS clients but with competitive throughput.

Resiliency: No single point of failure—CNodes are stateless and can fail without data loss. DBoxes use Storage Class Memory (SCM) for metadata durability and QLC flash for capacity. Data is erasure-coded across DBoxes. Trade-off: relies on NFS semantics (close-to-open by default), and the SCM write buffer in the DBoxes can become a bottleneck under sustained high-throughput writes.

# Filesystem Semantics Comparison

POSIX defines a set of filesystem behaviors that applications rely on—atomic operations, consistency guarantees, and locking primitives. Network filesystems make different trade-offs between full POSIX compliance and performance. Understanding these semantics tells you what your application can safely assume.

[1] NFSv3 supports reliable exclusive creation through CREATE(EXCLUSIVE); NFSv4 integrates open and lock state into the core protocol.
[2] NFS close-to-open: changes by one client visible to others only after close()+open(); see deep-dive section for details on edge cases
[3] NLM (Network Lock Manager) provides advisory locks; server crash can lose lock state
[4] Lustre LDLM (Distributed Lock Manager) provides strong cache coherency via lock callbacks
[5] Lustre locks can be revoked under contention, forcing clients to flush dirty data
[6] GPFS tokens grant read/write/lock permissions; distributed algorithm ensures consistency
[7] WekaFS maintains cache coherency across all clients via distributed consensus protocol
[8] VAST uses enhanced NFS (multipath, RDMA) but inherits NFS semantics; can configure stricter modes

# Key Implications

If your application requires strict POSIX semantics (e.g., databases with byte-range locking, build systems expecting consistent metadata), NFS's close-to-open model may surprise you. Lustre, GPFS, and WekaFS provide stricter guarantees but at the cost of lock traffic and potential contention. For read-mostly workloads or applications that don't share files between processes, NFS's relaxed model often suffices. For how each filesystem caches data and keeps it consistent—what lives in the page cache versus the client's own memory—see Caching Across Network Filesystems.

# Architecture, Scaling & Performance

Architecture determines how systems scale and where bottlenecks emerge. Single-server designs (NFS) are simple but limited by one machine's resources. Distributed designs (Lustre, GPFS, WekaFS) scale out but introduce complexity. Understanding these trade-offs helps you predict performance for your workload.

[8]  NetApp NFS performance guide TR-4067; typical enterprise server
[9]  Lustre Operations Manual 2.15; single MDT performance limits
[10] IBM GPFS Performance Tuning Guide; large enterprise deployment
[11] Weka technical documentation; AI/ML workload benchmarks
[12] VAST Data technical documentation; designed for AI-era workloads with high concurrency

# Workload Matching

Architecture dictates performance sweet spots. Match your workload to filesystem strengths:

• Deep learning training: Millions of small image files > WekaFS's distributed cache and metadata performance excel, while NFS would crawl under metadata load.
• Genomics pipelines: Terabyte-scale files, sequential I/O > Lustre's parallel data paths across multiple OSTs shine, delivering 100+ GB/s aggregate throughput.
• Mixed enterprise workloads: VMs, databases, home directories > GPFS or NFS offer good balance for varied access patterns without extreme specialization.
• HPC checkpointing: Periodic massive writes from thousands of ranks > Lustre (if you can tolerate load shedding) or GPFS (better reliability but lower peak throughput).

# Failure Modes & Overload Behavior

Systems fail. Clients misbehave. Loads spike. How filesystems handle these scenarios determines whether you experience graceful degradation or catastrophic failure. Understanding failure modes is essential for production deployments—it's not if things break, but when and how.

[13] NFS hang behavior infamous in HPC; "the NFS hang of death"
[14] Lustre load shedding: OST silently discards writes under overload; write() succeeds but data never reaches disk. Only detectable via OST logs (search for "ENOSPC" or drop messages). Catastrophic for applications assuming POSIX write semantics.
[15] GPFS recovery protocol: "Scalable cluster-wide failure recovery" (FAST '08)
[16] VAST CNodes are stateless; failure means clients reconnect to another CNode with no data loss

# Misbehaving Client Scenarios

What happens when a client crashes mid-write while holding locks?

• NFS: Server detects client timeout (typically 90s), releases locks, continues. Other clients may see partial writes if data wasn't flushed. NFSv4 improves with lease-based recovery.
• Lustre: LDLM has lock recovery protocol. If client crashes with dirty data, that data is lost (client-side caching). Lock timeout ~20s, other clients can proceed. Corruption possible if application didn't fsync().
• GPFS: Token manager detects node failure via heartbeat, forcibly revokes tokens, performs recovery. Strong consistency maintained. Recovery typically completes in seconds for small failures.
• WekaFS: Distributed consensus detects failure, invalidates client's cache, reassigns ownership. Replica ensures no data loss. Other clients continue with minimal disruption.
• VAST: NFS-based recovery—server detects client timeout and releases locks. CNodes are stateless, so no server-side dirty data to lose. Client reconnects to another CNode transparently. Data in DBoxes is erasure-coded and unaffected by client or CNode failures.

# Real-World Implications

Lustre's silent data loss: Under sustained write overload, Lustre OSTs can silently drop writes while reporting success to the application. Your write() returns successfully, but data never reaches disk. The only indication is cryptic messages in OST logs. For critical workloads, you must implement application-level checksumming or verification, and monitor OST logs for drop messages. This violates POSIX expectations and has caused data loss in production HPC systems.

NFS blocking: With hard-mount retry behavior, filesystem calls can remain blocked while the server is unreachable and can stall dependent workers. Treat that as an application-availability concern: isolate NFS-dependent work, use safe deadlines at application or workflow boundaries, monitor blocked operations, and define recovery behavior. Choose retry semantics from explicit correctness and data-handling requirements rather than treating a mount option as a simple consistency-versus-availability switch.

GPFS, WekaFS, and VAST: Provide better resilience with fast recovery and no silent data loss, but at higher cost. VAST trades some POSIX strictness for operational simplicity (NFS-based client access), while GPFS and WekaFS offer full POSIX compliance at greater complexity. All three require significant infrastructure investment.