> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/administration/managing-alluxio.md).

# Cluster and Membership Management

This guide covers cluster administration: production setup, scaling, upgrades, configuration, worker membership, and multi-tenancy/federation. For job submission, scheduling, and Coordinator HA, see [Job Service](https://documentation.alluxio.io/ee-ai-en/administration/managing-job-service.md). For per-worker configuration (storage, resources, JVM, and network), see [Worker Configuration](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md).

## 1. Cluster Setup and Lifecycle

### Production Setup

The basic configuration shown in the [Kubernetes Installation](https://documentation.alluxio.io/ee-ai-en/start/installing-on-kubernetes.md) guide is suitable for evaluation. For production deployments, apply the additional settings below for HA, resource tuning, persistent metadata, and worker identity.

### Label Nodes

A common practice is to assign dedicated nodes to each Alluxio component. This prevents resource contention between components (for example, etcd I/O interfering with worker cache I/O) and gives you predictable placement for capacity planning.

```shell
kubectl label nodes <coordinator-node> alluxio-role=coordinator
kubectl label nodes <worker-node-1> alluxio-role=worker
kubectl label nodes <worker-node-2> alluxio-role=worker
kubectl label nodes <worker-node-3> alluxio-role=worker
kubectl label nodes <etcd-node-1> alluxio-role=etcd
kubectl label nodes <etcd-node-2> alluxio-role=etcd
kubectl label nodes <etcd-node-3> alluxio-role=etcd
```

Worker pods have an anti-affinity rule by default — multiple worker pods will not be scheduled on the same node.

### Production `alluxio-cluster.yaml`

```yaml
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
metadata:
  name: alluxio-cluster
  namespace: alx-ns
spec:
  image: <PRIVATE_REGISTRY>/alluxio-enterprise
  imageTag: AI-3.9-17.1.0
  properties:
    alluxio.license: <YOUR_CLUSTER_LICENSE>
  coordinator:
    nodeSelector:
      alluxio-role: coordinator
    metastore:
      type: persistentVolumeClaim
      storageClass: "gp2"
      size: 4Gi
    resources:
      # Set requests equal to limits for Guaranteed QoS, same as the worker.
      limits:
        cpu: "8"
        memory: "16Gi"
      requests:
        cpu: "8"
        memory: "16Gi"
    jvmOptions:
      - "-Xmx8g"
      - "-Xms8g"
  worker:
    nodeSelector:
      alluxio-role: worker
    count: 3
    pagestore:
      size: 1000Gi
      reservedSize: 100Gi
    resources:
      # Set requests equal to limits so the worker runs in the Guaranteed
      # QoS class and is the last to be evicted under node pressure.
      limits:
        cpu: "8"
        memory: "24Gi"
      requests:
        cpu: "8"
        memory: "24Gi"
    jvmOptions:
      - "-Xmx12g"
      - "-Xms12g"
      - "-XX:MaxDirectMemorySize=12g"
  etcd:
    replicaCount: 3
    nodeSelector:
      alluxio-role: etcd
```

Key differences from the basic configuration:

* **Node selectors**: Pin each component to dedicated nodes to prevent resource contention and ensure predictable placement. See the label commands above.
* **Worker count**: number of workers depending on the target cache volume and target throughput. For post-deployment scaling, see [Scaling the Cluster](#scaling-the-cluster).
* **ETCD replicas**: 3 for quorum-based HA. Deploy on dedicated, stable nodes.
* **Resource limits and JVM options**: Explicitly set to prevent OOM. The container memory limit must exceed the sum of `-Xmx` and `-XX:MaxDirectMemorySize`. For both workers and the coordinator, set `requests` equal to `limits` — this places the pods in the [Guaranteed QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/), so they are the last to be evicted when the node is under memory or CPU pressure. A Burstable pod (requests < limits) can be evicted while still inside its limit if another pod exceeds its request, causing abrupt cache loss (worker) or scheduler disruption (coordinator).
* **Persistent metastore**: Coordinator metadata survives pod restarts.

Other important settings for production deployment:

* **License Management**: A cluster license is the simplest way to get started. For production environments, a **deployment license** is recommended. See [Appendix C: License Management](https://documentation.alluxio.io/pages/P92iY7CDvLx99wPxm2On#c.-license-management) for details on both options.
* **Hash Ring Configuration**: It is critical to configure the hash ring **before** deployment, as changes can be destructive. For detailed guidance, see [Hash Ring Pre-Deployment Configuration](#pre-deployment-configuration).
* **Worker Identity Persistence**: Configure `worker.systemInfo.hostPath` so workers rejoin with the same UUID after restarts. Without this, each restart adds stale `OFFLINE` entries to the hash ring, degrading cache hit rates. See [Restarting a Worker](#restarting-a-worker).
* **Heterogeneous Clusters**: If your cluster includes workers with different capacities, you must define a specific data distribution strategy. See [Heterogeneous Workers](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md#heterogeneous-workers) for configuration steps.
* **Worker Page Store**: The [page store](https://documentation.alluxio.io/pages/iRTxT4smG58AwFmCihOx#id-5.-worker-storage-the-page-store) is where each worker caches data. Key defaults and options:
  * **Default:** `type: hostPath`, `hostPath: /mnt/alluxio/pagestore`. The worker writes cache to the node's filesystem at that path. On multi-disk nodes, verify this lands on a data disk, not the system disk.
  * **Multi-disk nodes:** Set `pagestore.hostPath` explicitly to a data disk (e.g. `/mnt/data1/alluxio/pagestore`). See [Multi-Disk Configuration](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md#multi-disk-configuration).
  * **Persistent cache across pod restarts:** Use a PVC instead of hostPath. See [Configuring Page Store Location](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md#configuring-page-store-location).
  * **Sizing:** The `size` parameter sets the cache capacity; `reservedSize` allocates space for internal operations (temporary page writes, file metadata caching). Set `reservedSize` to \~10% of `size` (10–100 GiB) and ensure the total (size + reservedSize) fits within the worker's storage. For the cache size itself, target **at least 110–120% of your working set** (the data you expect to keep hot). Alluxio starts evicting at the high-watermark threshold (default 90% of `size`), so a 10–20% headroom above the working set avoids evicting in-use data during peak load jobs.
* **Advanced Configuration**: For resource and JVM tuning, see [Worker Configuration — Resource and JVM Tuning](https://documentation.alluxio.io/pages/MF8PfNU0P7q8PD9eh6xW#id-2.-resource-and-jvm-tuning). For other settings like external etcd, refer to [Appendix B: Advanced Configuration](https://documentation.alluxio.io/pages/P92iY7CDvLx99wPxm2On#b.-advanced-configuration).

### Running Multiple Clusters on Shared Nodes

If multiple Alluxio clusters are deployed across different namespaces on the same Kubernetes cluster, services from different clusters may be scheduled onto the same node, causing deployment failures. Label nodes to indicate which cluster they belong to:

```shell
kubectl label nodes <node-name> cluster=alluxio-a
kubectl label nodes <node-name> cluster=alluxio-b
```

Then specify the `nodeSelector` at the cluster level in each `alluxio-cluster.yaml`:

```yaml
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  image: <PRIVATE_REGISTRY>/alluxio-enterprise
  imageTag: AI-3.9-17.1.0
  nodeSelector:
    cluster: alluxio-a
```

### Cluster Lifecycle and Configuration

This section covers fundamental operations related to the cluster's lifecycle, such as scaling, upgrades, and dynamic configuration updates.

#### Scaling the Cluster

You can dynamically scale the number of Alluxio workers up or down. Because Alluxio uses a consistent hash ring to map files to workers, adding or removing workers reshards the ring — some files that were mapped to existing workers will now map to the new worker (or to different workers after scale-down). Those slots start empty, so **scale operations must be coordinated with running load jobs** to avoid coverage gaps.

The steps below are the **generic scaling flow**: you reshard the ring by changing the worker count, then re-warm the newly assigned slots passively with `job load`. A spike of cache misses is expected after new, empty workers join the cluster, and gradually self-heals as the load job fills the gap. If you cannot tolerate a cache-miss spike during scaling, follow [**Cache Rebalancing**](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md) instead — a separate, staged *stage → rebalance → commit* workflow that pre-positions data between workers *before* the topology change goes live, rather than back-filling from UFS afterward. It is its own end-to-end procedure, not a step within this one; use one approach or the other for a given operation, not both.

{% hint style="warning" %}
Always stop running load jobs before scaling. A load job running across a ring topology change may produce incomplete cache coverage — files assigned to the new worker after resharding will not have been loaded.
{% endhint %}

**Scale Up**

**Step 1: Stop any running load jobs.**

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# List running jobs
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job list --job-type LOAD --job-state RUNNING
# Expected: empty output if no load jobs are running

# Stop each running job by path
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <path> --stop
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio job list --job-type LOAD --job-state RUNNING
# Expected: empty output if no load jobs are running

bin/alluxio job load --path <path> --stop
```

{% endtab %}
{% endtabs %}

**Step 2: Increase the worker count and apply.**

Modify `alluxio-cluster.yaml` and increase `worker.count` (example: 2 → 3):

```shell
kubectl apply -f alluxio-cluster.yaml
```

```console
alluxiocluster.k8s-operator.alluxio.com/alluxio-cluster configured
```

**Step 3: Wait for all workers to become ready.**

```shell
kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker
```

```console
NAME                                          READY   STATUS    RESTARTS   AGE
alluxio-cluster-worker-58999f8ddd-cd6r2       1/1     Running   0          5m21s
alluxio-cluster-worker-58999f8ddd-rtftk       1/1     Running   0          4m21s
alluxio-cluster-worker-58999f8ddd-p6n59       1/1     Running   0          34s
```

Alternatively, wait with a timeout:

```shell
kubectl -n alx-ns wait pod -l app.kubernetes.io/component=worker \
  --for=condition=Ready --timeout=300s
```

**Step 4: Re-trigger data loading.**

After resharding, the new worker holds no cached data — client requests for files that now map to it are served from UFS until those files are loaded onto it. Data already cached on existing workers is **not deleted or migrated automatically**; those files become orphaned replicas that are no longer routed to and will evict naturally via LRU.

Re-warm the newly assigned slots with `job load --skip-if-exists`, which loads only the files missing from their new owner and leaves existing workers untouched:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <ufs-or-alluxio-path> --submit --skip-if-exists

# Monitor until SUCCEEDED
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <ufs-or-alluxio-path> --progress
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio job load --path <ufs-or-alluxio-path> --submit --skip-if-exists

# Monitor until SUCCEEDED
bin/alluxio job load --path <ufs-or-alluxio-path> --progress
```

{% endtab %}
{% endtabs %}

**Scale Down**

**Step 1: Stop any running load jobs** (same as scale-up Step 1 above).

**Step 2: Decrease the worker count and apply.**

```shell
kubectl apply -f alluxio-cluster.yaml
```

Wait for the removed worker pods to terminate:

```shell
kubectl -n alx-ns wait pod -l app.kubernetes.io/component=worker \
  --for=delete --timeout=120s
```

Scaling down removes workers from the hash ring. Their cached data becomes unreachable and will eventually be evicted. Files that were cached only on the removed workers will be served from UFS until re-loaded by passive caching or a new `job load`.

**To avoid that cache-miss spike,** relocate the leaving workers' cached data onto the surviving workers *before* removing them — stage the removals, rebalance, and commit — using a [Cache Rebalancing](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md) scale-in, then decrease `worker.count`.

**Step 3: Optionally deregister stale worker entries.**

In dynamic mode (default), `OFFLINE` entries are automatically cleaned up after `alluxio.worker.failure.detection.timeout`. In static mode, run `alluxio process remove-worker` for each decommissioned worker — see [Removing a Worker Permanently](#removing-a-worker-permanently).

#### Upgrading Alluxio

The upgrade process involves two main steps: upgrading the Alluxio Operator and then upgrading the Alluxio cluster itself.

**Step 1: Upgrade the Operator**

The operator is stateless and can be safely re-installed without affecting the running Alluxio cluster.

1. Obtain the new Docker images for the operator and the new Helm chart.
2. Uninstall the old operator and install the new one.

Uninstall the current operator:

```shell
helm -n alluxio-operator uninstall operator --wait
```

```console
release "operator" uninstalled
```

Replace the CRDs from the new Helm chart directory and create the new ones:

```shell
kubectl create -f alluxio-operator/crds
kubectl replace -f alluxio-operator/crds
```

```console
customresourcedefinition.apiextensions.k8s.io/alluxioclusters.k8s-operator.alluxio.com replaced
customresourcedefinition.apiextensions.k8s.io/clustergroups.k8s-operator.alluxio.com replaced
customresourcedefinition.apiextensions.k8s.io/collectinfoes.k8s-operator.alluxio.com replaced
customresourcedefinition.apiextensions.k8s.io/licenses.k8s-operator.alluxio.com replaced
customresourcedefinition.apiextensions.k8s.io/underfilesystems.k8s-operator.alluxio.com replaced
```

Install the new operator using your configuration file (update the image tag):

```shell
helm -n alluxio-operator install operator -f alluxio-operator.yaml --create-namespace .
```

**Step 2: Upgrade the Alluxio Cluster**

The operator will perform a rolling upgrade of the Alluxio components.

1. Upload the new Alluxio Docker images to your registry.
2. Update the `imageTag` in your `alluxio-cluster.yaml` to the new version.
3. Apply the configuration change.

Apply the updated cluster definition:

```shell
kubectl apply -f alluxio-cluster.yaml
```

```console
alluxiocluster.k8s-operator.alluxio.com/alluxio-cluster configured
```

Monitor the rolling upgrade process:

```shell
kubectl -n alx-ns get pod
```

```console
NAME                                          READY   STATUS     RESTARTS   AGE
alluxio-cluster-coordinator-0                 0/1     Init:0/2   0          7s
...
alluxio-cluster-worker-58999f8ddd-cd6r2       0/1     Init:0/2   0          7s
alluxio-cluster-worker-5d6786f5bf-cxv5j       1/1     Running    0          10m
```

Check the cluster status until it returns to 'Ready':

```shell
kubectl -n alx-ns get alluxiocluster
```

```console
NAME              CLUSTERPHASE   AGE
alluxio-cluster   Updating       10m
...
NAME              CLUSTERPHASE   AGE
alluxio-cluster   Ready          12m
```

Verify the new version is running:

```shell
kubectl -n alx-ns exec -it alluxio-cluster-coordinator-0 -- alluxio info version 2>/dev/null
```

```console
AI-3.9-17.1.0
```

During the rolling upgrade, workers are restarted in batches, such that the workers in the current batch must be fully ready before the next batch starts. The default batch size is 10% of the workers.

The number of workers in a batch can be reduced to minimize the interruption to running workloads during the period, at the cost of extending the period. To control the proportion or set the exact number of workers to restart, set the following in `alluxio-cluster.yaml`:

```yaml
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  worker:
    rollingUpdate:
      maxUnavailable: 1  # by default this is 10%, but in addition to setting a percentage value, it can also be set to an exact number
```

#### Updating Configuration

Many properties can be changed on a running cluster without a restart; see [Dynamic Configuration](https://documentation.alluxio.io/ee-ai-en/administration/dynamic-configuration.md) and the list in [Dynamic Properties](https://documentation.alluxio.io/ee-ai-en/reference/dynamic-properties.md). To change any other property, edit the cluster's ConfigMap and restart the components that read it.

1. **Find the ConfigMap** for your cluster.

   ```shell
   kubectl -n alx-ns get configmap
   ```

   ```console
   NAME                              DATA   AGE
   alluxio-cluster-alluxio-conf      4      7m48s
   ...
   ```
2. **Edit the ConfigMap** to modify `alluxio-site.properties`, `alluxio-env.sh`, etc.

   ```shell
   kubectl -n alx-ns edit configmap alluxio-cluster-alluxio-conf
   ```
3. **Restart components** to apply the changes.
   * **Coordinator:** `kubectl -n alx-ns rollout restart statefulset alluxio-cluster-coordinator`
   * **Workers:** `kubectl -n alx-ns rollout restart deployment alluxio-cluster-worker`
   * **DaemonSet FUSE:** `kubectl -n alx-ns rollout restart daemonset alluxio-fuse`
   * **CSI FUSE:** These pods must be restarted by exiting the application pod or by manually deleting the FUSE pod (`kubectl -n alx-ns delete pod <fuse-pod-name>`).

## 2. Membership and Worker Lifecycle

{% hint style="warning" %}
Hash ring settings should be defined during the initial cluster setup. Modifying these configurations on a running cluster is a destructive operation that will cause all cached data to be lost, as it changes how data is mapped to workers.
{% endhint %}

Alluxio represents worker membership through a consistent hash ring that maps data to workers. This chapter explains how workers join, leave, and rejoin the cluster, and how to configure the ring that governs those membership changes.

### Pre-Deployment Configuration

Set the following properties before first deployment:

| Property                                                                             | Default          | When to change                                                                                                                                                              |
| ------------------------------------------------------------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio.user.dynamic.consistent.hash.ring.enabled`                                  | `true` (dynamic) | Set to `false` (static) only if you need a stable ring view despite temporary worker unavailability — see [Configuring the Hash Ring Mode](#configuring-the-hash-ring-mode) |
| `alluxio.user.worker.selection.policy.consistent.hash.virtual.node.count.per.worker` | `2000`           | Rarely — only for very small or heavily imbalanced clusters. See [Adjusting Virtual Nodes](#adjusting-virtual-nodes-for-load-balancing)                                     |
| `alluxio.user.worker.selection.policy.consistent.hash.provider.impl`                 | `DEFAULT`        | Set to `CAPACITY` for heterogeneous worker clusters. See [Optimizing for Heterogeneous Workers](#optimizing-for-heterogeneous-workers)                                      |

{% tabs %}
{% tab title="Kubernetes (Operator)" %}
Set under `.spec.properties` in `alluxio-cluster.yaml`:

```yaml
spec:
  properties:
    alluxio.user.dynamic.consistent.hash.ring.enabled: "true"
    alluxio.user.worker.selection.policy.consistent.hash.provider.impl: DEFAULT
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}
Set in `conf/alluxio-site.properties` (applies to client, worker, and coordinator processes):

```properties
alluxio.user.dynamic.consistent.hash.ring.enabled=true
alluxio.user.worker.selection.policy.consistent.hash.provider.impl=DEFAULT
```

{% endtab %}
{% endtabs %}

### Hash Ring Configuration

#### Configuring the Hash Ring Mode

The consistent hash ring operates in two base modes — **dynamic** (default) or **static**. Static mode can additionally run as a **versioned** hash ring for controlled, cache-miss-free rebalancing; see [Versioned Hash Ring](#versioned-hash-ring) below.

In **dynamic mode** (default), the hash ring includes only online workers. When a worker goes offline, it is removed from the ring after the liveness timeout, and its virtual nodes are redistributed to live workers. This is the best choice for most deployments — the ring adapts automatically to permanent topology changes such as scaling or node replacement.

In **static mode**, the hash ring retains all registered workers regardless of their online status. This mode is designed for **planned short-term maintenance** — rolling software upgrades, hardware servicing, or brief restarts where workers are expected to rejoin quickly. By keeping offline workers in the ring, Alluxio avoids resharding cached data: when the worker comes back, it reclaims its original ring slots and all its cached data is immediately routable again. The trade-off is that during the downtime window, requests that hash to the offline worker fall through to UFS.

To configure the mode, set the `alluxio.user.dynamic.consistent.hash.ring.enabled` property. Set it to `true` for dynamic mode (the default) or `false` for static mode.

#### Versioned Hash Ring

The versioned hash ring is an **extension of static mode** for operators who need to change cluster membership *without* the burst of cache misses a normal topology change causes. It is the mechanism behind [Cache Rebalancing](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md).

Because it must not auto-drop workers, it builds on **static** mode (`alluxio.user.dynamic.consistent.hash.ring.enabled=false`) and adds staged membership: the cluster keeps a **current** ring that clients route on and an editable **staging** ring, and a membership change takes effect only when you explicitly **commit** it — after a rebalance job has pre-positioned the affected data on its new owners. It also requires etcd-backed coordination and `alluxio.user.consistent.hash.ring.versioning.enabled=true`.

**Enable it only when you need** [**Cache Rebalancing**](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md#when-to-use-it) — cache-miss-free scale-out/scale-in or planned node replacement. It adds real operational overhead (the stage → rebalance → commit workflow), so if you don't need that, **prefer the default dynamic mode** for its simplicity. See Cache Rebalancing for [how it works](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md#how-it-works), [configuration](https://documentation.alluxio.io/ee-ai-en/cache/cache-rebalancing.md#configuration), and the runbooks, and the [Membership REST API](https://documentation.alluxio.io/ee-ai-en/reference/rest-api.md#membership) for the endpoints.

#### Adjusting Virtual Nodes for Load Balancing

To ensure an even distribution of data and I/O requests, Alluxio uses virtual nodes. Each worker is mapped to multiple virtual nodes on the hash ring, which helps to balance the load more effectively across the cluster.

You can adjust the number of virtual nodes per worker by configuring the `alluxio.user.worker.selection.policy.consistent.hash.virtual.node.count.per.worker` property (default: `2000`). Adjusting this value can help fine-tune load distribution, especially in clusters with diverse workloads or a small number of workers.

#### Optimizing for Heterogeneous Workers

By default, the consistent hashing algorithm assumes that all workers have equal capacity. In clusters with heterogeneous workers (e.g., different storage capacities or network speeds), you can enable capacity-based allocation for more balanced resource utilization. This ensures that workers with more storage handle a proportionally larger share of data.

To enable this, set the `alluxio.user.worker.selection.policy.consistent.hash.provider.impl` property to `CAPACITY`. The default value is `DEFAULT`, which allocates an equal number of virtual nodes to each worker.

For the worker-side YAML configuration (labeling nodes, `workerGroups`), see [Heterogeneous Workers](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md#heterogeneous-workers).

#### Worker Liveness Detection

Each worker maintains active communication with etcd. If a worker fails to communicate with etcd within the timeout period, it is considered offline:

```properties
# increase if workers are slow to reconnect after transient network issues (default: 15s)
alluxio.worker.failure.detection.timeout=30s
```

In dynamic mode, an `OFFLINE` worker's virtual nodes are removed from the hash ring after this timeout and redistributed to live workers. In static mode, the `OFFLINE` entry remains in the ring — requests hashing to it fall through to UFS until the worker rejoins or is explicitly removed.

#### Client Worker List Refresh

Clients maintain a local snapshot of the worker list and refresh it periodically from etcd, every `workerListRefreshIntervalMs` of the client–worker timeout policy (default 45 seconds). Reduce it for faster failover; increase it to lower etcd read load. It can be changed on a running cluster without a restart; see [Client–Worker Timeouts](https://documentation.alluxio.io/pages/MF8PfNU0P7q8PD9eh6xW#id-4.-clientworker-timeouts).

`alluxio.user.worker.list.refresh.interval` only sets the starting value when the policy is first created. Changing it afterwards has no effect.

After adding or removing workers, clients will reflect the change within one refresh interval. To force immediate propagation during incident recovery, restart the client-side process or reduce this interval temporarily.

### Worker Membership and Lifecycle

Worker membership changes affect where data is mapped and whether cached data remains reachable. The procedures below cover checking membership, adding or removing workers, restarting workers, and restoring cache coverage.

#### Checking Worker Status

To see a list of all registered workers and their current status:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- alluxio info nodes
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio info nodes
```

{% endtab %}
{% endtabs %}

```console
WorkerID                              Host               Status
e87b4097-b7f8-48a4-a388-3197124ebbe7  ip-172-16-10-12    ONLINE
a1f23c45-d678-90ab-cdef-123456789012  ip-172-16-10-13    ONLINE
```

**Diagnosing Hash Ring Bloat from OFFLINE Entries**

A healthy cluster should show one `ONLINE` entry per configured worker. If `alluxio info nodes` shows more total entries than expected, or the `ONLINE` count is lower than the worker count, stale `OFFLINE` entries are accumulating in etcd.

**Impact** (static mode): In static mode (`alluxio.user.dynamic.consistent.hash.ring.enabled=false`), `OFFLINE` entries remain in the ring indefinitely. With stale entries present, a proportional fraction of hash lookups land on `OFFLINE` nodes and fall through to UFS — turning cache hits into direct S3/GCS reads at native object-store speeds.

In dynamic mode (default), `OFFLINE` entries are automatically removed from the ring after `alluxio.worker.failure.detection.timeout` (see [Worker Liveness Detection](#worker-liveness-detection)), so ring bloat does not persist. However, each restart with a new UUID still remaps virtual nodes, making previously cached data temporarily unreachable.

Verify the ring is healthy — ONLINE count should equal your configured worker count:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- alluxio info nodes | grep -c ONLINE
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio info nodes | grep -c ONLINE
```

{% endtab %}
{% endtabs %}

To remove a stale `OFFLINE` entry from etcd, get its UUID from the `alluxio info nodes` output and run:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio process remove-worker -n <STALE_WORKER_UUID>
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio process remove-worker -n <STALE_WORKER_UUID>
```

{% endtab %}
{% endtabs %}

#### Adding or Removing Workers

New workers register in etcd and join the hash ring automatically at startup. Removing a worker redistributes its hash ring portion to remaining workers, causing a temporary increase in cache misses while data is re-fetched from UFS.

{% tabs %}
{% tab title="Kubernetes (Operator)" %}
Adjust the `worker.count` in `alluxio-cluster.yaml` and apply — see [Scaling the Cluster](#scaling-the-cluster) for the full procedure.
{% endtab %}

{% tab title="Docker / Bare-Metal" %}
Start a new worker process on the target host (docker: `docker run ... alluxio/alluxio-enterprise worker`; bare-metal: `bin/alluxio process start worker`). To remove, stop the worker process; see [Removing a Worker Permanently](#removing-a-worker-permanently) for the deregistration step.
{% endtab %}
{% endtabs %}

#### Removing a Worker Permanently

When decommissioning a node, stop the worker first and then explicitly deregister it from etcd so its entry does not remain as a stale `OFFLINE` node.

**Step 1: Stop the worker.**

{% tabs %}
{% tab title="Kubernetes (Operator)" %}
Scale down the worker count in `alluxio-cluster.yaml` and apply:

```shell
kubectl apply -f alluxio-cluster.yaml
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}
Stop the worker process on the decommissioned host (docker: `docker stop alluxio-worker`; bare-metal: `bin/alluxio process stop worker`).
{% endtab %}
{% endtabs %}

**Step 2: Deregister the worker from etcd.** Get the worker's UUID from `alluxio info nodes`, then run `alluxio process remove-worker`:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- alluxio info nodes
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio process remove-worker -n <WORKER_UUID>
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio info nodes
bin/alluxio process remove-worker -n <WORKER_UUID>
```

{% endtab %}
{% endtabs %}

In dynamic mode, skipping Step 2 is usually safe — the `OFFLINE` entry is automatically purged after `alluxio.worker.failure.detection.timeout`. In static mode, Step 2 is required to prevent the stale entry from staying in the hash ring indefinitely.

#### Restarting a Worker

When a worker restarts, it is temporarily marked as offline. With identity persistence configured, the worker rejoins with the same UUID — preserving its ring position, cached data, and load distribution.

Without identity persistence, each restart generates a new UUID with different ring slots, causing cache misses and potential ring bloat. See [Configuring the Hash Ring Mode](#configuring-the-hash-ring-mode) and [Diagnosing Hash Ring Bloat from OFFLINE Entries](#diagnosing-hash-ring-bloat-from-offline-entries) for details.

**Persisting Worker Identity**

{% tabs %}
{% tab title="Kubernetes (Operator)" %}
Set `worker.systemInfo.hostPath` in `alluxio-cluster.yaml` before first deployment:

```yaml
spec:
  worker:
    useExternalId: false
    systemInfo:
      hostPath: /mnt/alluxio/system-info
```

{% endtab %}

{% tab title="Docker" %}
Pre-create an empty file on the host before first start, then mount it as a volume. Alluxio writes the UUID into it on startup, and it survives container recreation:

```shell
sudo mkdir -p /etc/alluxio
sudo touch /etc/alluxio/worker_identity
```

Add to the worker `docker run` command:

```shell
-v /etc/alluxio/worker_identity:/opt/alluxio/conf/worker_identity
```

{% hint style="warning" %}
Do **not** use `-v` without pre-creating the file on the host first. If the host path does not exist, Docker creates a directory at that path instead of a file, causing Alluxio to fail with `IOException: Is a directory`.
{% endhint %}

For the full Docker setup guide, see [Appendix B: Worker Identity](https://documentation.alluxio.io/pages/xiYDWQw61suARXg2DRbX#b.-worker-identity).
{% endtab %}

{% tab title="Bare-Metal" %}
Set `alluxio.worker.identity.uuid.file.path` to a path that survives reboots:

```properties
alluxio.worker.identity.uuid.file.path=/etc/alluxio/worker_identity
```

The file is created automatically on first start. To manually pin an existing worker's UUID (e.g., when migrating to a persistent path), first get the current UUID from `alluxio info nodes`, then write it to the configured path.
{% endtab %}
{% endtabs %}

{% hint style="warning" %}
If a worker is migrated to a different host, copy the identity file to the same path on the new host before starting the worker. Without it, the worker registers with a new UUID and all cached data from the previous identity becomes unreachable.
{% endhint %}

#### Cache Recovery After Worker Restart

**With identity persistence configured**, a restarted worker rejoins the hash ring under the same UUID and its previously cached pages remain accessible. However, the restarted worker may lack data that was **loaded onto other workers while it was offline** — during the outage those files were routed to workers that temporarily held this worker's ring slots, so the recovered worker has never cached them. Once it rejoins, those files hash back to it but are absent from its local cache.

To fill those gaps without redundant re-loading:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# Confirm the worker has rejoined the ring
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio info nodes | grep ONLINE
# Expected: N entries all showing ONLINE (N = configured worker count)

# Re-trigger loading — skips files already cached on any worker
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <ufs-or-alluxio-path> --submit --skip-if-exists

# Monitor until SUCCEEDED
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <ufs-or-alluxio-path> --progress
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# Confirm the worker has rejoined the ring
bin/alluxio info nodes | grep ONLINE
# Expected: N entries all showing ONLINE (N = configured worker count)

# Re-trigger loading
bin/alluxio job load --path <ufs-or-alluxio-path> --submit --skip-if-exists

# Monitor until SUCCEEDED
bin/alluxio job load --path <ufs-or-alluxio-path> --progress
```

{% endtab %}
{% endtabs %}

`--skip-if-exists` ensures that files already cached on healthy workers are not re-fetched from UFS. Only the files that now map to the recovered worker but are not yet cached there will be loaded.

**Without identity persistence**, the worker registers with a new UUID, occupying different ring slots. All data cached under the old UUID becomes unreachable, and a full reload is required to restore cache coverage. This is why identity persistence is a production requirement — see [Persisting Worker Identity](#persisting-worker-identity).

## 3. Multi-Tenancy and Federation

For large-scale enterprise deployments, Alluxio provides advanced features for multi-tenancy and cluster federation. This allows multiple teams and business units to share data infrastructure securely and efficiently while simplifying administrative overhead.

The reference architecture below features an API Gateway that centrally handles authentication and authorization across multiple Alluxio clusters.

### Core Concepts

#### Authentication

Alluxio integrates with external enterprise identity providers like **Okta**. When a user logs in, the provider authenticates them and generates a [**JSON Web Token (JWT)**](https://auth0.com/docs/secure/tokens/access-tokens/access-token-profiles). This JWT is then sent with every subsequent request to the Alluxio API Gateway to verify the user's identity.

#### Authorization

Once a user is authenticated, Alluxio uses an external policy engine, [**Open Policy Agent (OPA)**](https://www.openpolicyagent.org/docs/latest/#running-opa), to determine what actions the user is authorized to perform. Administrators can write fine-grained access control policies in OPA's declarative language, **Rego**, to control which users can access which resources. The API Gateway queries OPA for every request to ensure it is authorized.

#### Multi-Tenancy and Isolation

Alluxio enforces isolation between tenants to ensure security and prevent interference. This is achieved through:

* **User Roles:** Defining different roles with specific access levels and permissions.
* **Cache Isolation:** Assigning tenant-specific cache configurations, including quotas, TTLs, and eviction policies, ensuring one tenant's workload does not negatively impact another's.

### Cluster Federation

For organizations with multiple Alluxio clusters (e.g., across different regions or for different business units), federation simplifies management. A central **Management Console** provides a single pane of glass for:

* Cross-cluster monitoring and metrics.
* Executing operations across multiple clusters simultaneously.
* Centralized license management for all clusters.

### Example Workflow: Updating a Cache Policy

This workflow demonstrates how the components work together:

1. **Authentication:** A user logs into the **Management Console**, which redirects them to **Okta** for authentication. Upon success, Okta issues a JWT.
2. **Request Submission:** The user uses the console to submit a request to change a cache TTL. The request, containing the JWT, is sent to the **API Gateway**.
3. **Authorization:** The API Gateway validates the JWT and queries the **OPA Policy Engine** to check if the user has permission to modify cache settings for the target tenant.
4. **Execution:** If the request is authorized, the API Gateway forwards the command to the coordinator of the relevant Alluxio cluster, which then applies the new TTL policy.
