Decentralized Architecture & Worker Management

Overview of the Decentralized Architecture

Alluxio uses a decentralized, decentralized architecture to achieve high availability and massive scalability. Instead of relying on a central master for metadata, Alluxio distributes this responsibility across all the workers in the cluster using a consistent hashing algorithm.

What is a Hash Ring?

At the core of consistent hashing is the hash ring. Imagine a circular ring where we place both the Alluxio workers and the data (or more accurately, the hash of the data's path).

1. Workers on the Ring: Each worker node in the cluster is assigned one or more positions on this ring.

2. Data on the Ring: To find out which worker is responsible for a file, the Alluxio client calculates a hash of the file's path and places it on the same ring.

3. Finding the Owner: The client then travels clockwise around the ring from the data's position until it finds the first worker. That worker is the "owner" of the data and is responsible for its metadata and cache.

This approach is what allows the architecture to be "decentralized." The client can determine the correct worker to talk to with a simple calculation, without needing to ask a central authority.

Key Benefits

This design provides several key benefits:

• No Single Point of Failure (SPOF): The system can tolerate the loss of worker nodes without experiencing a full outage. When a worker is removed, only the keys it owns need to be remapped to the next worker on the ring.

• Linear Scalability: You can scale the cluster's metadata and data capacity simply by adding more workers. Adding a new worker only requires a small number of keys to be remapped from its neighbors on the ring.

• Low Latency: Clients can locate and access data with minimal network hops.

Alluxio uses a distributed key-value store, etcd, to manage cluster membership and track which workers are online, which keeps the view of the hash ring consistent across all clients.

Configuring Worker Behavior

You can fine-tune the behavior of the workers and the consistent hashing ring by setting properties in alluxio-site.properties.

Consistent Hashing Settings

Property	Description	Default
alluxio.user.worker.selection.policy.consistent.hash.virtual.node.count.per.worker	The number of virtual nodes assigned to each worker on the hash ring. More nodes provide better load distribution but increase memory overhead.	2000
alluxio.user.worker.selection.policy.consistent.hash.provider.impl	Set to CAPACITY to allocate virtual nodes based on each worker's storage capacity. This is useful for heterogeneous clusters.	NODE_ID
alluxio.user.dynamic.consistent.hash.ring.enabled	If true (dynamic), only online workers are on the ring. If false (static), all registered workers are included, even if offline. The dynamic setting is recommended for most use cases.	true

Worker Membership and Identity

Property	Description	Default
alluxio.worker.membership.manager.type	The backend for managing worker membership.	ETCD
alluxio.etcd.endpoints	A comma-separated list of connection URIs for your etcd cluster.
alluxio.worker.identity.uuid.file.path	Path to a file that stores the worker's unique identity. Persisting this allows a worker to rejoin the cluster after a restart with its cached data intact.	/mnt/alluxio/system-info/worker_identity
alluxio.worker.failure.detection.timeout	The time a worker can be unresponsive before it's considered offline by etcd.	2m
alluxio.user.worker.list.refresh.interval	How often the Alluxio client fetches the updated worker list from etcd.	2m

Managing Workers (Operations)

These commands allow you to perform day-to-day operational tasks on your cluster.

Checking Worker Status

To see a list of all registered workers and their current status (online or offline), run:

bin/alluxio info nodes

Adding a New Worker

To add a new worker to the cluster:

1. Install the Alluxio software on the new node.

2. Ensure the alluxio-site.properties file is configured to point to your etcd cluster.

3. Start the worker process. It will automatically register itself in etcd and join the consistent hashing ring.

When a new worker joins, it takes over a portion of the hash ring from existing workers. Initially, requests for data in its range will result in cache misses that are served from the UFS, but the worker will begin caching data immediately.

Removing a Worker Permanently

If you need to decommission a worker permanently, follow these steps:

1. Shut down the worker process on the target node to ensure it's offline.

2. Get the Worker ID: Run bin/alluxio info nodes to list all workers and find the ID of the one you want to remove.

3. Remove the worker: Use the process remove-worker command with the worker's ID.

   bin/alluxio process remove-worker -n <worker_id>

4. Verify removal: Run bin/alluxio info nodes again to confirm the worker is no longer in the list of registered nodes.

Important: Removing a worker is a permanent action that will cause its portion of the hash ring to be redistributed to other workers. This may lead to a temporary increase in cache misses as the cluster rebalances.

Restarting a Worker (Temporary Maintenance)

If you restart a worker (e.g., for maintenance), it will be temporarily marked as offline. As long as its identity is preserved via the alluxio.worker.identity.uuid.file.path setting, it will rejoin the cluster and be responsible for the same portion of the hash ring it managed previously. Its cached data will remain valid and available as soon as it comes back online.

During the restart, ongoing requests to that worker will fail over to other resiliency mechanisms, such as UFS fallback.