Apache ZooKeeper

• Apache ZooKeeper is a distributed coordination service for managing large sets of hosts. It provides a centralized, reliable, and consistent framework to facilitate distributed applications by handling configuration management, synchronization, naming registries, and group services.

• ZooKeeper maintains a hierarchical namespace similar to a file system, where each node is called a znode. It runs as a cluster (ensemble) of servers to provide high availability and consistency using the Zab consensus protocol.

What are Znodes?

• Znodes are the fundamental data units in ZooKeeper, arranged hierarchically like files and directories.
• Each znode can:
• Store data (up to 1 MB).
• Contain child znodes forming subtrees.
• Znodes maintain metadata such as:
• Version numbers (data, children, ACL changes).
• Creation and modification timestamps.
• Access Control Lists (ACLs) for security.
• Types of znodes:
• Persistent: Remain until explicitly deleted.
• Ephemeral: Exist as long as the client's session is active; deleted automatically upon session expiration. Used for leader election and session tracking.
• Sequential: Named with a unique, increasing sequence number (can combine with persistent or ephemeral). Useful for ordered coordination like locks.

Use Cases of ZooKeeper with Examples

1. Distributed Locks

ZooKeeper provides primitives to implement distributed locks ensuring mutual exclusion across distributed processes:

• Clients create ephemeral sequential znodes under a lock path.
• The client with the smallest sequence number holds the lock.
• Other clients watch their predecessor node and acquire the lock when that node disappears.
• Example: Several processes simultaneously updating a shared database record can coordinate using a ZooKeeper-based lock to prevent concurrent conflicts.

2. Leader Election

ZooKeeper facilitates leader election to ensure only one node controls certain critical operations in a distributed system:

• Nodes attempt to create an ephemeral sequential znode under a leader election path.
• The node with the smallest sequence number becomes the leader.
• If the leader node fails, its ephemeral node is deleted, triggering followers to elect a new leader.
• Example: In Hadoop NameNode high-availability setups, ZooKeeper elects the active NameNode among multiple candidates.

3. Configuration Management

ZooKeeper stores centralized configuration data accessible to distributed nodes:

• Configurations (e.g., connection strings, settings) are stored in znodes.
• Clients watch these znodes and dynamically reload configurations on changes.
• Example: All nodes in a Kafka cluster can fetch and watch broker configurations from ZooKeeper to keep settings synchronized.

4. Service Discovery

Services register themselves by creating znodes under well-known paths, enabling clients to discover live instances:

• Service instances create ephemeral znodes to signal their availability.
• Clients watch the service node for changes to discover new or failed instances.
• Example: Microservice architectures use ZooKeeper for services to find and route to each other reliably.

5. Distributed Queues and Barrier Synchronization

ZooKeeper can implement coordination patterns like distributed queues or barriers by leveraging znodes and their orderings:

• Sequential znodes help maintain ordering.
• Ephemeral znodes help track node availability.

How ZooKeeper Handles Notifications

• Clients register watches on znodes during reads.
• If a znode’s data or children change, all clients with registered watches are automatically notified asynchronously.
• Watches trigger only once; clients must re-register if they want continual change notifications.
• This enables real-time reactive behavior without the need for polling.

Pros of ZooKeeper

• Simplifies complex distributed coordination tasks.
• Provides strong consistency and reliable atomic updates.
• High read throughput—you can read from any node.
• Automatic failure detection through ephemeral nodes.
• Mature ecosystem with wide adoption.

Cons of ZooKeeper

• Leader bottleneck for writes can limit performance.
• Designed for small metadata, not large data storage.
• Adding nodes and ensemble scaling requires careful management.
• Not a multi-datacenter solution for geo-distributed clusters out of the box.
• Operational overhead for setup and maintenance.

Summary

ZooKeeper is an essential distributed coordination service providing hierarchical, consistent znodes backed by a consensus protocol (Zab). It's widely used for:

• Distributed locks for safe resource sharing.
• Leader election for fault-tolerant control.
• Centralized configuration management.
• Service discovery in dynamic systems.
• Coordination patterns like queues and barriers.

Its notification system with watches enables efficient event-driven updates to clients, making it a cornerstone for building reliable distributed applications.