> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/administration/managing-etcd.md).

# etcd Management

Alluxio keeps its durable cluster state in etcd: the mount table, configuration entities, the deployment license, and — when versioning is on — the hash ring member lists. Cached data is not in etcd; it lives on worker disk, which is why etcd work can usually be done without going cold.

On Kubernetes the Operator provisions etcd for you. This page covers what to do after that: pointing at an etcd you run yourself, tuning the bundled one, moving to a different etcd, and rebuilding when one is beyond repair.

For sizing and quorum rules before you install, see [Prerequisites](/ee-ai-en/ai-3.8-15.1.x/start/prerequisites.md#etcd). For etcd across availability zones, see [Multiple AZ](/ee-ai-en/ai-3.8-15.1.x/high-availability/multi-az.md).

## Using an External etcd

To run against an etcd cluster you manage rather than the one the Operator deploys:

```yaml
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    enabled: false
  properties:
    alluxio.etcd.endpoints: http://external-etcd:2379
    # If using TLS for ETCD, add the following:
    # alluxio.etcd.tls.enabled: "true"
```

On a cluster that is already running and holding state, do not just apply this — the new etcd starts empty and the mount table does not follow. Use [Migrating to a Different etcd](#migrating-to-a-different-etcd) instead.

For securing the connection, see [TLS](/ee-ai-en/ai-3.8-15.1.x/administration/security/securing-alluxio-with-tls.md#securing-etcd-communication).

## Customizing the Bundled etcd

The fields under `spec.etcd` follow the [Bitnami etcd Helm chart](https://github.com/bitnami/charts/blob/main/bitnami/etcd/values.yaml). For example, to pin etcd pods to particular zones with [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity):

```yaml
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - antarctica-east1
              - antarctica-west1
```

## Migrating to a Different etcd

Copy the durable keys to the new etcd, repoint the cluster, restart. The cache stays hot throughout — it is on worker disk, not in etcd.

Use this when moving from the bundled etcd to a managed one, or replacing an etcd cluster. If the old etcd is already unrecoverable, this procedure does not apply; see [Rebuilding etcd](#rebuilding-etcd).

### What gets copied

Only three things are durable and have to move:

| Key                        | Holds                                      |
| -------------------------- | ------------------------------------------ |
| `/alluxio/CONF/<cluster>/` | Mount table and all configuration entities |
| `/LE/license/<cluster>`    | Deployment license                         |
| `/LE/version/<cluster>`    | License version                            |

Everything else rebuilds itself. Worker registrations under `/ServiceDiscovery/` and the license instance records under `/LE/instances/` and `/LE/status/` are held on etcd leases, so they expire and are re-created when the processes restart.

The exception is the hash ring member lists, which do not self-heal — but only if you turned ring versioning on. See [If ring versioning is enabled](#if-ring-versioning-is-enabled).

{% hint style="warning" %}
etcd values are binary. Every copy below round-trips through `base64` so the bytes survive exactly — a plain `get | put` appends a newline and corrupts the value.
{% endhint %}

### If either etcd uses TLS

A managed etcd usually does, and the commands below assume plaintext. Give `etcdctl` the certificates for whichever end needs them — both, if the source is also TLS — on every invocation:

```shell
etcdctl --endpoints=$DST \
  --cacert=/path/to/etcd-ca.crt \
  --cert=/path/to/etcd-client.crt \
  --key=/path/to/etcd-client.key \
  endpoint health
```

Keeping that readable is worth an alias:

```shell
DSTCTL="etcdctl --endpoints=$DST --cacert=/path/to/etcd-ca.crt \
  --cert=/path/to/etcd-client.crt --key=/path/to/etcd-client.key"
```

Alluxio needs the same trust material in Step 6, alongside `alluxio.etcd.endpoints`:

```yaml
    alluxio.etcd.tls.enabled: "true"
    alluxio.etcd.tls.ca.cert: /path/to/etcd-ca.crt
    alluxio.etcd.tls.client.cert: /path/to/etcd-client.crt
    alluxio.etcd.tls.client.key: /path/to/etcd-client-key-pkcs8.pem
```

The endpoint becomes `https://`, and **Alluxio requires the client key in PKCS#8** — a format `etcdctl` does not care about, so the same key file will not serve both. See [TLS](/ee-ai-en/ai-3.8-15.1.x/administration/security/securing-alluxio-with-tls.md#securing-etcd-communication) for the conversion and for `alluxio.etcd.username` / `alluxio.etcd.password` if the target also uses password auth.

### Prerequisites

Run the `etcdctl` commands from any host or pod that has `etcdctl` and can reach **both** endpoints. They use the v3 API, which is the default from etcdctl v3.4 on; on anything older, export `ETCDCTL_API=3` first. The `alluxio` and `kubectl` commands run against the Alluxio pods.

Set these once. Substitute your own namespace, cluster name, and endpoints:

```shell
# Source: the etcd Service endpoint, which fronts all HA members.
# Not a single member — if your managed etcd exposes only per-member
# addresses, list them all comma-separated.
SRC=http://alluxio-cluster-etcd.alx-ns:2379

# Target: the new etcd Service endpoint
DST=http://new-etcd:2379

# Cluster name = <namespace>-<AlluxioCluster name>. It prefixes every key.
CL=alx-ns-alluxio-cluster
```

Confirm `CL` rather than assuming it:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio conf get alluxio.cluster.name
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio conf get alluxio.cluster.name
```

{% endtab %}
{% endtabs %}

```console
alx-ns-alluxio-cluster
```

The value must equal `$CL`. If it does not, every key path below is wrong.

Check that `etcdctl` reaches both ends before going further:

```shell
etcdctl --endpoints=$SRC endpoint health
etcdctl --endpoints=$DST endpoint health
```

```console
http://alluxio-cluster-etcd.alx-ns:2379 is healthy: successfully committed proposal: took = 1.9ms
http://new-etcd:2379 is healthy: successfully committed proposal: took = 2.4ms
```

### Step 1: Record a baseline

Both outputs must be unchanged after the cutover.

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# Worker identities
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio info nodes

# Mount URIs — compare byte for byte afterwards, including the scheme
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio mount list
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio info nodes
bin/alluxio mount list
```

{% endtab %}
{% endtabs %}

Save both outputs. Step 7 compares against them.

```console
WorkerId        Address          Status
worker-1        10.0.1.11:29999  ONLINE
worker-2        10.0.1.12:29999  ONLINE
```

### Step 2: Check the target etcd

The target may already hold unrelated keys — that is fine, Alluxio only reads and writes under its own prefixes. What must not exist is state for **this** `cluster.name`.

```shell
for P in /alluxio/CONF/$CL/ /LE/license/$CL /LE/version/$CL /DHT/$CL /ServiceDiscovery/$CL; do
  echo -n "$P -> "; etcdctl --endpoints=$DST get "$P" --prefix --keys-only | grep -c .
done
```

```console
/alluxio/CONF/alx-ns-alluxio-cluster/ -> 0
/LE/license/alx-ns-alluxio-cluster -> 0
/LE/version/alx-ns-alluxio-cluster -> 0
/DHT/alx-ns-alluxio-cluster -> 0
/ServiceDiscovery/alx-ns-alluxio-cluster -> 0
```

A non-zero count means a previous or aborted install already used this etcd with this `cluster.name`. Clear those prefixes, or choose a different `cluster.name`, before copying — otherwise you layer onto stale state.

### Step 3: Copy the durable keys

```shell
# Idempotent — re-running overwrites the same keys with the same bytes
{ etcdctl --endpoints=$SRC get /alluxio/CONF/$CL/ --prefix --keys-only | grep -v '^$'; \
  echo "/LE/license/$CL"; echo "/LE/version/$CL"; } \
  | while read -r K; do
      [ -z "$K" ] && continue
      etcdctl --endpoints=$SRC get "$K" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/' | base64 -d \
        | etcdctl --endpoints=$DST put "$K" >/dev/null && echo "copied: $K"
    done
```

```console
copied: /alluxio/CONF/alx-ns-alluxio-cluster/mount/s3a:%2F%2Fbucket
copied: /alluxio/CONF/alx-ns-alluxio-cluster/cachefilter/default
copied: /LE/license/alx-ns-alluxio-cluster
copied: /LE/version/alx-ns-alluxio-cluster
```

One `copied:` line per key. A key that prints nothing did not transfer — Step 4 catches it.

### Step 4: Verify the copy

Every line must say `MATCH`.

```shell
vfy(){ a=$(etcdctl --endpoints=$SRC get "$1" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/'); \
       b=$(etcdctl --endpoints=$DST get "$1" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/'); \
       [ "$a" = "$b" ] && echo "MATCH    $1" || echo "MISMATCH $1"; }

{ etcdctl --endpoints=$SRC get /alluxio/CONF/$CL/ --prefix --keys-only | grep -v '^$'; \
  echo "/LE/license/$CL"; echo "/LE/version/$CL"; } \
  | while read -r K; do [ -z "$K" ] && continue; vfy "$K"; done
```

```console
MATCH    /alluxio/CONF/alx-ns-alluxio-cluster/mount/s3a:%2F%2Fbucket
MATCH    /alluxio/CONF/alx-ns-alluxio-cluster/cachefilter/default
MATCH    /LE/license/alx-ns-alluxio-cluster
MATCH    /LE/version/alx-ns-alluxio-cluster
```

A `MISMATCH` means that key did not survive the round-trip. Re-copy it before going on. Do not continue with any line reading `MISMATCH`.

### Step 5: Dump the source for rollback

```shell
etcdctl --endpoints=$SRC get "" --prefix -w json > source-keyspace-full.json
```

Keep this. It is what [Rollback](#rollback) falls back to if the old etcd volumes are gone.

### Step 6: Repoint and restart

In the `AlluxioCluster` CR, set `etcd.enabled: false` and add `alluxio.etcd.endpoints`. Do not change `metadata.name`, the mount, or `alluxio.cluster.name` — the cluster name is what every copied key is prefixed with.

```shell
# Idempotent. The Operator deletes the bundled etcd but keeps its PVCs,
# then restarts the coordinator and workers.
kubectl apply -f alluxio-cluster.yaml -n alx-ns

# Wait rather than watch
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=coordinator --timeout=300s
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=worker --timeout=300s
```

```console
pod/alluxio-cluster-coordinator-0 condition met
pod/alluxio-cluster-worker-0 condition met
pod/alluxio-cluster-worker-1 condition met
```

FUSE pods must roll too. DaemonSet FUSE pods restart with the cluster; CSI-provisioned ones do not, so restart them and then the application pods that mount them.

```shell
kubectl get pods -n alx-ns -l 'app.kubernetes.io/component in (fuse, csi-fuse)'
# Expected: every pod Running, AGE younger than the apply above
```

On Docker or bare metal there is no CR: set `alluxio.etcd.endpoints` in `alluxio-site.properties` on every node, then restart the coordinator, the workers, and the FUSE processes.

### Step 7: Verify

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio info nodes
# Expected: same WorkerIds as Step 1. Addresses may differ; identities may not.

kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio mount list
# Expected: byte-identical to Step 1, scheme included
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio info nodes
bin/alluxio mount list
```

{% endtab %}
{% endtabs %}

Then confirm the cache survived. Read the two counters off a worker, re-read a dataset that was cached before the migration, and read them again — see [Monitoring](/ee-ai-en/ai-3.8-15.1.x/administration/monitoring-alluxio.md) for the endpoints:

```shell
kubectl -n alx-ns exec alluxio-cluster-worker-0 -- \
  curl -s http://localhost:30000/metrics/ \
  | grep -E 'alluxio_cache_(hit|miss)_calls_total'
# Expected after the re-read: the hit count rises by roughly one per file
# and the miss count does not move. Any rise in misses means that data is
# being fetched from the UFS again — the cache did not survive.
```

### If ring versioning is enabled

Ring member lists do not rebuild themselves, so they need copying too. This only applies when `alluxio.user.consistent.hash.ring.versioning.enabled` is `true`; it defaults to `false`, in which case skip this.

```shell
# Empty output means versioning is off — skip
etcdctl --endpoints=$SRC get /DHT/$CL/currentMembers

# Otherwise copy all three lists
for M in currentMembers stagingMembers historicMembers; do
  etcdctl --endpoints=$SRC get /DHT/$CL/$M -w json | sed -E 's/.*"value":"([^"]+)".*/\1/' | base64 -d \
    | etcdctl --endpoints=$DST put /DHT/$CL/$M >/dev/null 2>&1 && echo "copied: /DHT/$CL/$M"
done
```

See [Hash Ring and Worker Lifecycle](/ee-ai-en/ai-3.8-15.1.x/administration/managing-ring.md) for what these lists do.

### Rollback

**If the old etcd PVCs still exist** — the normal case, since Step 6 keeps them — revert the CR (`etcd.enabled: true`, drop the `alluxio.etcd.endpoints` line) and apply. The Operator re-adopts the old PVCs by name.

```shell
kubectl apply -f alluxio-cluster.yaml -n alx-ns
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=coordinator --timeout=300s
# Expected: "condition met". Then re-run Step 7 against the old etcd.
```

**If they are already deleted**, stand up an empty etcd and replay the durable keys from the Step 5 dump into it, then point the CR at that:

```shell
python3 - source-keyspace-full.json http://fresh-etcd:2379 <<'PY'
import json, sys, base64, subprocess
d = json.load(open(sys.argv[1])); dst = sys.argv[2]
KEEP = ("/alluxio/CONF/", "/LE/license/", "/LE/version/")
for kv in d["kvs"]:
    if int(kv.get("lease", 0) or 0):
        continue                      # leased keys rebuild themselves
    k = base64.b64decode(kv["key"]).decode()
    if not k.startswith(KEEP):
        continue
    v = base64.b64decode(kv.get("value", "") or "")
    subprocess.run(["etcdctl", "--endpoints", dst, "put", k], input=v, check=True)
    print("restored", k)
PY
```

### Cleanup

Only after Step 7 passes. Deleting these volumes is the point of no return for the simple rollback above.

```shell
kubectl get pvc -n alx-ns | grep etcd
kubectl delete pvc data-alluxio-cluster-etcd-0 data-alluxio-cluster-etcd-1 \
  data-alluxio-cluster-etcd-2 -n alx-ns
```

## Rebuilding etcd

Alluxio tolerates an etcd outage for a grace period (typically 24 hours) without disrupting I/O, so an etcd pod that restarts cleanly needs no action. Rebuild only when the cluster is unrecoverable and its data cannot be copied out — otherwise [migrate](#migrating-to-a-different-etcd) instead, which keeps your state.

{% hint style="danger" %}
Rebuilding discards cluster state, including the mount table. Expect to re-create mounts afterwards.
{% endhint %}

1. Shut down the Alluxio cluster: `kubectl delete -f alluxio-cluster.yaml`
2. Delete the etcd PVCs: `kubectl -n alx-ns delete pvc -l app.kubernetes.io/component=etcd`
3. Clear etcd data on the nodes: log into each Kubernetes node that hosted an etcd pod and delete the contents of the host path directory used by the etcd PV.
4. Recreate the cluster: `kubectl create -f alluxio-cluster.yaml`. The Operator provisions a new, empty etcd cluster.
5. Re-mount your UFS paths. If you manage mounts with the `UnderFileSystem` CRD they are restored for you; otherwise re-add them with `alluxio mount add --path <path> --ufs-uri <uri>`.

A three-replica etcd cluster only survives a node loss if the replicas are on distinct nodes. Co-located replicas lose quorum together.

## Related

* [Prerequisites — etcd](/ee-ai-en/ai-3.8-15.1.x/start/prerequisites.md#etcd) — node count and quorum rules before you install
* [Multiple AZ](/ee-ai-en/ai-3.8-15.1.x/high-availability/multi-az.md) — independent, shared, and external etcd topologies across zones
* [TLS](/ee-ai-en/ai-3.8-15.1.x/administration/security/securing-alluxio-with-tls.md#securing-etcd-communication) — securing the etcd connection
* [Monitoring](/ee-ai-en/ai-3.8-15.1.x/administration/monitoring-alluxio.md) — etcd availability alerting
* [Job Service](/ee-ai-en/ai-3.8-15.1.x/administration/managing-job-service.md#optional-dedicated-etcd-cluster-for-job-scheduling) — a separate etcd for job scheduling
* [Installing on Kubernetes](/ee-ai-en/ai-3.8-15.1.x/start/installing-on-kubernetes.md) — etcd symptoms seen during installation