GFS — Google File System

What you'll learn

• Why Google Built It: The Workload Assumptions

  In 2003, Google was processing the web. They had crawlers generating terabytes of data, MapReduce jobs reading gigabytes sequentially, and BigTable needing append-heavy write patterns. Traditional distributed file systems (NFS, AFS) were…

• Files Are Huge

  Files are multi-GB. A web crawl produces 100GB files. A MapReduce output is 50GB. The median file size is not 4KB—it's hundreds of megabytes. This changes everything. If your files are huge, you can use large block sizes (chunks in GFS t…

• Reads Are Mostly Sequential and Large

  When a MapReduce worker reads input, it streams through the entire file. It doesn't seek randomly to byte 1,234,567. It reads sequentially, megabytes at a time. This means: - Prefetching works great - Caching file data at clients is less…

• Writes Are Mostly Appends

  Google's applications don't overwrite files. They write once, append to them (logs, intermediate MapReduce results, crawl outputs), and then read them. Random overwrites are rare. This is fundamentally different from a traditional file s…

• Component Failures Are the Norm, Not the Exception

  Google built GFS on commodity Linux machines. Disks fail. Machines reboot. Network switches flake. In a cluster of 1,000 machines, something is always broken. MTBF (mean time between failures) for a single machine might be 3 years; for a…

• High Sustained Bandwidth > Low Latency

  Google cares about throughput: "Can I read 1GB/sec across 100 files?" They don't care much about latency: "Can I read one 4KB block in 5ms?" The workload is batch-oriented. MapReduce jobs run for hours. If a single read takes 50ms instea…

• Contrast with POSIX Assumptions

  POSIX file systems assume: - Files are small (KBs to MBs) - Random access is common (lseek + read) - Overwrite semantics (write to any offset) - Strong consistency (after write returns, all readers see the new data) - Low-latency metadat…

• Core Abstractions

• Files and Chunks

  A GFS file is logically a byte sequence, just like any file. But physically, it's split into fixed-size chunks. Each chunk is 64MB (default). Why 64MB? 1. Reduces metadata size: A 1TB file is 16,000 chunks, not 268 million 4KB blocks. Th…

• Why Lazy?

  Simpler than eager deletion: - Eager deletion (immediately free all chunks) requires the master to track down all replicas and issue delete commands synchronously. If a chunkserver is temporarily offline, you have to retry, log failures,…

• Orphan Chunk Detection via Heartbeats

  How does the master tell a chunkserver which chunks are orphans? It doesn't send a special "delete chunk X" command. Instead: 1. Chunkserver heartbeat: Every 10-30 seconds, each chunkserver sends a heartbeat to the master. In the heartbe…

• Snapshots: Copy-on-Write via Reference Counting

  GFS supports snapshots: making a copy of a file or directory tree almost instantaneously, with negligible overhead.

• Use Case

  MapReduce jobs want to "checkpoint" intermediate state: "Save a copy of all files in /intermediate/ at this instant, so if the job crashes, I can restart from this snapshot." Traditional copy would take minutes (copy TBs). GFS snapshot t…

• Implementation: Copy-on-Write

  1. Client calls snapshot("/src", "/snap"). 2. Master revokes all outstanding write leases on files in /src (waits for them to expire or forces chunkservers to release them). 3. Master creates a new directory entry /snap that points to th…

• Replica Rebalancing: How GFS Fills New Chunkservers

  When Google adds new chunkservers to a GFS cluster (to scale storage capacity), the master doesn't immediately flood the new machines with writes. Instead, it gradually rebalances replicas to the new servers. Here's how: The master runs…

• Namespace Management Without Inodes

  Traditional filesystems (like ext3 or xfs) use inodes: fixed-size metadata structures that store file attributes (size, timestamps, pointers to data blocks). GFS doesn't have inodes. Instead: The namespace is a hash map: - Key: full path…

• What GFS Got Wrong: Lessons and Colossus

  GFS powered Google for years, but it had limits. By the late 2000s, Google was hitting them:

• Single Master Bottleneck

  As Google's data grew, so did the metadata: - Billions of files → hundreds of millions of chunks → GBs of metadata in RAM. - Master CPU became a bottleneck: handling tens of thousands of metadata ops per second (opens, creates, list dire…

• Replication Overhead

  3× replication is simple but expensive: - 1PB of data requires 3PB of disks. - Network bandwidth for writes: 3× (client pushes to 3 replicas). Colossus fix: Reed-Solomon erasure coding. Instead of 3 full replicas, Colossus uses (e.g.) 9…

• Small File Performance

  GFS's 64MB chunk size is terrible for small files: - A 1KB file consumes a full 64MB chunk (wasted space—though lazy allocation mitigates this). - Worse, if many clients read that 1KB file, all reads hit the same 3 chunkservers (hotspot)…

• Master Failover Time

  Tens of seconds to recover a master is slow. During that time, writes are stalled (reads can go to shadow masters). For latency-sensitive applications (like serving user queries), this is unacceptable. Colossus fix: Faster failover with…

• The Consistency Model Is a Tax

  GFS's weak consistency worked for MapReduce and batch workloads, but it's a pain for applications that want "write once, read consistently." Developers had to sprinkle checksums, IDs, and deduplication logic everywhere. Here's the develo…

• GFS's Legacy

  Despite these issues, GFS was a massive success: - It scaled Google from TBs to PBs of storage. - It proved that a distributed FS could be built on commodity hardware with acceptable performance. - It influenced Hadoop (HDFS is a near-cl…

• Interview Takeaways: What to Remember

  When discussing GFS in an interview, emphasize the design trade-offs: 1. Single master for simplicity: The master handles metadata only (not data), so it's not a bandwidth bottleneck. This design avoids distributed consensus for namespac…

• Further Reading

  If you want to dive deeper: - Original GFS paper (Ghemawat, Gobioff, Leung, 2003): The source of truth. 15 pages, very readable. - Colossus paper/talks: Google hasn't published a formal Colossus paper, but engineers have given talks at F…

• Final Thoughts

  GFS is a product of its time and workload. In 2003, Google needed to store TBs of crawled web data on thousands of commodity machines. POSIX semantics didn't fit. Strong consistency was too expensive. Random writes were rare. So they bui…