HDFS — Hadoop Distributed File System

What you'll learn

• HDFS vs GFS — Lineage

  HDFS was directly inspired by the 2003 GFS paper from Google. The core architecture is nearly identical, with components mapping one-to-one: | GFS Term | HDFS Term | Notes | |------------------|-----------------|-------------------------…

• Key Departures from GFS

  Despite the architectural similarity, HDFS diverged in important ways: 1. Namespace Structure: - GFS: Flat namespace with path-like keys (like S3). - HDFS: Real hierarchical directory tree (POSIX-like), stored as an inode structure in th…

• Core Abstractions

• Files and Blocks

  HDFS files are split into blocks (default 128MB). Each block is: - Identified by a Block ID (64-bit integer, monotonically increasing per NameNode). - Versioned with a Generation Stamp (GS) to handle pipeline failures during writes. If a…

• Replication

  Default replication factor = 3. Configurable per file (e.g., critical logs might use 5×, temporary MapReduce data might use 1×). Replica Placement: 1. First replica: On the client's node (if client is a DataNode) or a random node in the…

• Block Pools (HDFS 2.0+)

  With Federation, each NameNode manages a Block Pool (namespace + blocks). DataNodes store blocks from multiple pools, tagged with a BP- prefix in the directory structure: This allows multiple independent namespaces to share the same Data…

• Architecture — NameNode + DataNodes + Client

  HDFS has a master-worker architecture with a single active NameNode (or an active+standby pair in HA mode) and many DataNodes (hundreds to thousands).

• NameNode (The Metadata Master)

  The NameNode is the brain of HDFS. It stores all metadata in RAM for fast lookups. There is no metadata database; the NameNode's heap is the source of truth at runtime. In-Memory State: 1. Namespace (INode Tree): - A hierarchical tree of…

• DataNodes (The Storage Workhorses)

  DataNodes store blocks as regular files on local disks (e.g., ext4, xfs). Typical configurations: - 12-24 HDDs per DataNode (JBOD, not RAID). - 10GbE or 25GbE NICs. - 64-128GB RAM (for block scanner, caching). Block Storage: - Each block…

• Client (The Data Plane)

  The HDFS client is a library (Java DFSClient, or libhdfs C wrapper) embedded in applications (MapReduce, Spark, HBase). Key Point: The client talks directly to DataNodes for data. The NameNode is never in the data path. This is the same…

• Read Path

  The read path is straightforward and optimized for sequential throughput. Step-by-Step: 1. Client calls open(path): - Client sends RPC to NameNode: getBlockLocations(path, offset, length). - NameNode checks permissions, looks up the file…

• Write Path — Pipeline of DataNodes

  The write path is the most complex part of HDFS. It uses a pipeline of DataNodes to stream data, achieving high throughput while writing 3 replicas.

• Overview

  Unlike reads (which go directly client → DataNode), writes go: in a sequential pipeline. Data flows through the chain, and ACKs flow back in reverse. This design saves network bandwidth (client sends data once, not three times) and overl…

• Detailed Write Sequence

  Step 1: Create the File 1. Client sends RPC to NameNode: create(path, replication, blockSize, permissions). 2. NameNode checks: - Parent directory exists and is writable by the client. - File doesn't already exist (or overwrite is allowe…

• hflush vs hsync

  - hflush() (a.k.a. sync() in older versions): - Flushes all buffered packets down the pipeline. - DataNodes write to their OS buffer cache (not necessarily disk). - Guarantees visibility: After hflush(), new readers will see the data. -…

• Pipeline Recovery

  If a DataNode fails mid-write (e.g., DN2 crashes), the client must recover the pipeline without losing data. Recovery Steps: 1. Client detects failure (TCP timeout or error ACK). 2. Client closes the current pipeline. 3. Client increment…

• Block Placement Policy

  The default block placement policy is "one local, two remote (same rack)". Here's the exact algorithm:

• First Replica

  - If the client is a DataNode, place the first replica locally (zero-cost write). - Else, place it on a random DataNode in the client's rack. - If the client's rack is unknown or empty, pick a random DataNode in the cluster.

• Second Replica

  - Pick a different rack (uniformly random among racks with available capacity). - Pick a random DataNode in that rack.

• Third Replica

  - Pick a different DataNode in the same rack as the second replica.

• Fourth+ Replicas (if replication > 3)

  - Pick random DataNodes, avoiding racks/nodes already used. Example (replication=3): - Client is on rack1/host5. - First replica → rack1/host5 (local). - Second replica → rack2/host9 (different rack). - Third replica → rack2/host12 (same…

• Why This Policy?

  Durability: - Survives any single node failure (2 replicas remain). - Survives any single rack failure (1 replica remains). - Does not survive two rack failures (but that's rare in practice; many HDFS clusters have 2-5 racks). Write Band…

• NameNode Metadata Durability

  The NameNode's in-memory state is ephemeral. If the NameNode crashes, the RAM is lost. To recover, HDFS persists metadata to disk.

• FsImage (Filesystem Image)

  - A checkpoint of the namespace (INode tree + file→block mappings). - Binary format (protobuf-based in modern HDFS). - Stored as fsimage (where txid is the transaction ID). - Example: - Does not include block→DataNode locations (those ar…

• EditLog (Write-Ahead Log)

  - Every metadata mutation is first written to the EditLog before updating in-memory state. - Stored as edits. - Operations: - OPADD: Create a file or directory. - OPADDBLOCK: Allocate a block to a file. - OPCLOSE: Close a file (mark it i…

• Checkpoint Process (Secondary NameNode)

  In non-HA mode, the NameNode never creates FsImage itself (too slow, blocks metadata operations). Instead, a Secondary NameNode (a separate process, often on a different machine) does it: Steps: 1. Secondary NameNode fetches the latest F…

• Startup Sequence

  When the NameNode starts: 1. Load the latest fsimage. 2. Replay all edits files from txid+1 to the end. 3. Reconstruct the in-memory INode tree and file→block mappings. 4. Enter Safe Mode (read-only): - Wait for DataNodes to send block r…

• NameNode HA with Quorum Journal Manager (QJM)

  The NameNode is a single point of failure in non-HA mode. If it crashes, the entire HDFS cluster is offline until it restarts (and replays the EditLog, which can take minutes for large clusters). HDFS 2.0 introduced High Availability (HA…

• Architecture

  - Active NameNode: Handles all client requests, writes to EditLog. - Standby NameNode: Reads the EditLog, keeps its in-memory state in sync, checkpoints the FsImage. Does not serve client requests (except for monitoring RPCs). - JournalN…

• EditLog Sharing with QJM

  Write Path (Active NameNode): 1. Active NameNode writes each edit to its local buffer. 2. Active NameNode sends the edit to all JournalNodes in parallel (async RPC). 3. Quorum rule: If a majority of JournalNodes ACK (e.g., 2 out of 3), t…

• Fencing and Split-Brain

  Split-brain is the nightmare scenario: two NameNodes both think they're Active, both serve client writes, EditLogs diverge → data loss/corruption. HDFS prevents this with: 1. Fencing: The Standby-becoming-Active must fence the old Active…

• Consistency Semantics

  HDFS has strong consistency for the namespace (directory tree) and single-writer / many-readers for files.

• Single-Writer

  - Only one client can write to a file at a time (enforced by leases). - No concurrent writers (unlike GFS's record-append). - This simplifies consistency: no "undefined regions" from interleaved writes.

• Visibility on Close

  - While a file is open for writing, readers see committed blocks (blocks that have been hflush()'d or hsync()'d). - After close(), the file is immutable and fully visible to all readers immediately. Example: Reader (concurrent):

• Directory Operations

  - Atomic: rename(), mkdir(), delete() are atomic (single EditLog entry). - Linearizable: All clients see the same order of operations (thanks to the EditLog's sequential log). Example: From this moment, all clients see /user/bar, not /tm…

• Contrast with GFS

  - GFS: Supports concurrent record-appends to the same file. Writers might see duplicates or padding. Readers see a "defined" region (at least once) but not necessarily "consistent" (same bytes for all readers) until the file is closed. -…

• Failure Handling

  HDFS is designed to handle frequent hardware failures (disk, node, rack) without data loss.

• DataNode Failure

  Detection: - NameNode expects a heartbeat every 3 seconds. - If no heartbeat for 10 minutes (configurable, dfs.namenode.heartbeat.recheck-interval), the DataNode is marked dead. Response: 1. NameNode removes the DataNode from all block→D…

• Disk Failure

  DataNodes use JBOD (just a bunch of disks), not RAID. If a disk fails: 1. DataNode detects the failure (I/O errors, S.M.A.R.T. warnings). 2. DataNode stops using the disk, removes it from the volume list. 3. DataNode sends a heartbeat to…

• Block Corruption

  Detection: - DataNode block scanner: Periodically (every few weeks) scans all blocks, verifies checksums. If mismatch → log error, report to NameNode. - Client read: Verifies checksums on-the-fly. If mismatch → report to NameNode, retry…

• NameNode Failure

  Non-HA Mode: - NameNode crashes → entire HDFS cluster is unavailable (clients can't read or write). - Admin must restart the NameNode (can take minutes to replay EditLog and rebuild block maps from block reports). HA Mode: - Active NameN…

• Network Partition

  Scenario: Active NameNode is isolated from ZooKeeper, but alive. Response: 1. ZKFC on the Active NameNode loses the ZooKeeper session. 2. ZKFC fences the Active NameNode (shuts it down) to avoid split-brain. 3. Standby NameNode's ZKFC ac…

• Lease Recovery

  If a client crashes while writing a file (lease held, file not closed): 1. Lease times out (60 seconds soft limit, 60 minutes hard limit). 2. NameNode triggers lease recovery: - NameNode picks the first DataNode in the pipeline (DN1). -…

• Balancer

  Over time, HDFS clusters become imbalanced: - New DataNodes are added (empty). - Writes are unevenly distributed (hot files → hot DataNodes). - Decommissioning removes DataNodes → replicas redistributed unevenly. Imbalance causes: - Some…

• HDFS Balancer

  The Balancer is an offline tool (hdfs balancer) that moves blocks between DataNodes to achieve uniform disk usage across the cluster. Algorithm: 1. Balancer queries the NameNode for disk usage per DataNode. 2. Balancer computes a target…

• Safemode

  When the NameNode starts up or recovers from a failure, it enters Safemode — a read-only state where clients can read metadata and files, but no writes or modifications are allowed. Purpose: Safemode protects against premature writes tha…

• Balancer Daemon

  Over time, disk usage across DataNodes becomes uneven due to: - New DataNodes added (empty disks). - Uneven write patterns (hot files → certain DataNodes fill up faster). - Decommissioning (replicas redistributed, but not evenly). Proble…

• HDFS Balancer

  The Balancer is a standalone tool (hdfs balancer) that redistributes blocks to achieve uniform disk utilization across the cluster. How It Works: 1. Queries the NameNode for disk usage per DataNode. 2. Computes the average utilization (e…

• HDFS Router (Router-Based Federation, RBF)

  To provide a unified namespace across federated NameNodes, HDFS 3.0 introduced the Router-Based Federation (RBF). Architecture: Router is a stateless proxy that: 1. Accepts client RPCs (same interface as NameNode). 2. Looks up the path i…

• When to Use Federation

  - Large namespaces: If you have billions of files, a single NameNode's RAM is insufficient. Federation lets you split the namespace (e.g., by team or dataset). - High RPC load: If clients hit the NameNode's RPC limit (~100k ops/sec), Fed…

• Topology Configuration

  Admins configure rack topology via a topology script (dfs.network.topology.script.file.name): The NameNode calls this script to determine which rack each DataNode belongs to. Default: If no script is configured, all DataNodes are in /def…

• Rack-Aware Replication Recap

  Default policy (replication=3): 1. First replica: Client's node (if client is a DataNode) or random node in client's rack. 2. Second replica: Different rack (to survive rack failure). 3. Third replica: Same rack as second (to save cross-…

• Multiple Rack Replication (replication > 3)

  For replication=4: - First 3 replicas follow the standard policy. - Fourth replica: Random node/rack, avoiding existing nodes/racks. For replication=5: - Same, but try to place the 5th replica on a 3rd rack (if enough racks exist). High-…

• Erasure Coding (HDFS 3.0)

  Problem: 3× replication wastes 200% storage overhead. For cold data (archival logs, backups), this is expensive. Solution: Erasure Coding (EC) reduces overhead to ~50% while maintaining similar durability.

• How Erasure Coding Works

  Erasure coding splits a file into k data blocks and m parity blocks (total n = k + m). You can lose any m blocks and still reconstruct the file. Common schemes: - RS(6,3): 6 data blocks + 3 parity blocks = 9 total. Can tolerate 3 failure…

• HDFS EC Implementation

  Storage: - A file with EC policy RS(6,3) is split into block groups (6 data blocks + 3 parity blocks = 9 blocks). - Each block is 1/6th the size of a replicated block (e.g., 128MB / 6 ≈ 21MB per block). - Blocks are stored on 9 different…

• When to Use EC

  - Cold data: Archival logs, backups, historical datasets (infrequent reads, durability matters more than speed). - Large files: EC has per-block-group overhead; small files ( 3 blocks lost → data is permanently lost (cannot reconstruct).…

• Interview Takeaways

  If you're discussing HDFS in an interview, these are the key points that show you understand the design: 1. HDFS is not POSIX: - No random writes (append-only, or write-once). - No hard links or symbolic links (in early HDFS; added later…