etcd Management

Alluxio keeps its durable cluster state in etcd: the mount table, configuration entities, the deployment license, and — when versioning is on — the hash ring member lists. Cached data is not in etcd; it lives on worker disk, which is why etcd work can usually be done without going cold.

On Kubernetes the Operator provisions etcd for you. This page covers what to do after that: pointing at an etcd you run yourself, tuning the bundled one, recovering a single stuck member, moving to a different etcd, and rebuilding when one is beyond repair.

For sizing and quorum rules before you install, see Prerequisites. For etcd across availability zones, see Multiple AZ.

Using an External etcd

To run against an etcd cluster you manage rather than the one the Operator deploys:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    enabled: false
  properties:
    alluxio.etcd.endpoints: http://external-etcd:2379
    # If using TLS for ETCD, add the following:
    # alluxio.etcd.tls.enabled: "true"

On a cluster that is already running and holding state, do not just apply this — the new etcd starts empty and the mount table does not follow. Use Migrating to a Different etcd instead.

For securing the connection, see TLS.

Customizing the Bundled etcd

The fields under spec.etcd follow the Bitnami etcd Helm chart. For example, to pin etcd pods to particular zones with node affinity:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - antarctica-east1
              - antarctica-west1

Setting spec.etcd.affinity at all replaces the chart's entire default affinity, including the pod anti-affinity that spreads replicas across nodes — not just the node affinity you added. See Preventing repeat quorum loss before doing this on a running cluster.

Recovering a Single etcd Member

Use this when the cluster is otherwise healthy — quorum holds, so Alluxio keeps working — but one etcd pod will not come back on its own. It is far less invasive than migrating or rebuilding: no data is touched, and the mount table survives.

etcd's PVCs are ordinarily node-pinned local volumes, so a stuck member is almost always a scheduling problem, not corruption — the pod cannot follow its data to a different node. Two situations produce the identical symptom (pod stuck, not Ready) with very different fixes.

Diagnose

kubectl -n alx-ns describe pod alluxio-cluster-etcd-1 | grep -A5 Events

Warning  FailedScheduling  ...  0/5 nodes are available: 1 node(s) had untolerated taint
                                {node.kubernetes.io/unschedulable: }, 4 node(s) didn't
                                match Pod's node affinity/selector.

Find the node the member's volume is pinned to, and check whether it is cordoned:

PV=$(kubectl -n alx-ns get pvc data-alluxio-cluster-etcd-1 -o jsonpath='{.spec.volumeName}')
kubectl get pv "$PV" -o jsonpath='{.spec.nodeAffinity}'; echo
kubectl get nodes

{"required":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/hostname","operator":"In","values":["node-7"]}]}]}}

NAME     STATUS                     ROLES    AGE
node-7   Ready,SchedulingDisabled   <none>   240d

SchedulingDisabled means the node is cordoned. Cordoning only blocks new scheduling — it does not evict the pod already there, and it does not let the StatefulSet reschedule this member anywhere else, because its data only exists on node-7.

If the node is cordoned but otherwise healthy

kubectl uncordon node-7

The pod schedules back onto its own node and rejoins with its existing data. Nothing else to do.

If the node is gone for good

Quorum survives losing one of three members — do not lose a second one while doing this.

# 1. Remove the lost member's volume
kubectl -n alx-ns delete pvc data-alluxio-cluster-etcd-1

# 2. Delete the pod so the StatefulSet reschedules it. This needs a healthy
#    node that is not already running an etcd member — see anti-affinity below.
kubectl -n alx-ns delete pod alluxio-cluster-etcd-1

# 3. The chart has no hook that deregisters a member when its pod is deleted,
#    so its old entry is still registered and blocks the new pod from
#    rejoining cleanly. Remove it from a healthy member:
kubectl -n alx-ns exec alluxio-cluster-etcd-0 -- etcdctl member list
kubectl -n alx-ns exec alluxio-cluster-etcd-0 -- etcdctl member remove <stale-member-id>

# 4. The new pod provisions a fresh, empty volume on its new node and rejoins
kubectl -n alx-ns exec alluxio-cluster-etcd-0 -- etcdctl endpoint health --cluster
# Expected: 3 endpoints, all "healthy"

Preventing repeat quorum loss

The bundled chart spreads replicas across nodes by default — a soft preference, not a guarantee — and it is easy to lose without noticing: setting spec.etcd.affinity at all, as in the zone-pinning example above, replaces the entire affinity block and silently drops that default. Two etcd pods sharing a node means one node failure loses quorum outright.

Check current placement:

kubectl -n alx-ns get pod -l app.kubernetes.io/component=etcd -o wide
# Expected: the NODE column has three distinct values

If you set spec.etcd.affinity yourself, make sure it still spreads replicas:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/component: etcd
          topologyKey: kubernetes.io/hostname
      # any nodeAffinity you also need goes in this same block

requiredDuringSchedulingIgnoredDuringExecution is a hard requirement — a pod stays Pending rather than share a node. Only use it if you have at least as many eligible nodes as etcd replicas.

Migrating to a Different etcd

Copy the durable keys to the new etcd, repoint the cluster, restart. The cache stays hot throughout — it is on worker disk, not in etcd.

Use this when moving from the bundled etcd to a managed one, or replacing an etcd cluster. If the old etcd is already unrecoverable, this procedure does not apply; see Rebuilding etcd.

What gets copied

Only three things are durable and have to move:

Key	Holds
/alluxio/CONF/<cluster>/	Mount table and all configuration entities
/LE/license/<cluster>	Deployment license
/LE/version/<cluster>	License version

Everything else rebuilds itself. Worker registrations under /ServiceDiscovery/ and the license instance records under /LE/instances/ and /LE/status/ are held on etcd leases, so they expire and are re-created when the processes restart.

The exception is the hash ring member lists, which do not self-heal — but only if you turned ring versioning on. See If ring versioning is enabled.

etcd values are binary. Every copy below round-trips through base64 so the bytes survive exactly — a plain get | put appends a newline and corrupts the value.

If either etcd uses TLS

A managed etcd usually does, and the commands below assume plaintext. Give etcdctl the certificates for whichever end needs them — both, if the source is also TLS — on every invocation:

etcdctl --endpoints=$DST \
  --cacert=/path/to/etcd-ca.crt \
  --cert=/path/to/etcd-client.crt \
  --key=/path/to/etcd-client.key \
  endpoint health

Keeping that readable is worth an alias:

DSTCTL="etcdctl --endpoints=$DST --cacert=/path/to/etcd-ca.crt \
  --cert=/path/to/etcd-client.crt --key=/path/to/etcd-client.key"

Alluxio needs the same trust material in Step 6, alongside alluxio.etcd.endpoints:

    alluxio.etcd.tls.enabled: "true"
    alluxio.etcd.tls.ca.cert: /path/to/etcd-ca.crt
    alluxio.etcd.tls.client.cert: /path/to/etcd-client.crt
    alluxio.etcd.tls.client.key: /path/to/etcd-client-key-pkcs8.pem

The endpoint becomes https://, and Alluxio requires the client key in PKCS#8 — a format etcdctl does not care about, so the same key file will not serve both. See TLS for the conversion and for alluxio.etcd.username / alluxio.etcd.password if the target also uses password auth.

Prerequisites

Run the etcdctl commands from any host or pod that has etcdctl and can reach both endpoints. They use the v3 API, which is the default from etcdctl v3.4 on; on anything older, export ETCDCTL_API=3 first. The alluxio and kubectl commands run against the Alluxio pods.

Set these once. Substitute your own namespace, cluster name, and endpoints:

# Source: the etcd Service endpoint, which fronts all HA members.
# Not a single member — if your managed etcd exposes only per-member
# addresses, list them all comma-separated.
SRC=http://alluxio-cluster-etcd.alx-ns:2379

# Target: the new etcd Service endpoint
DST=http://new-etcd:2379

# Cluster name = <namespace>-<AlluxioCluster name>. It prefixes every key.
CL=alx-ns-alluxio-cluster

Confirm CL rather than assuming it:

Kubernetes (Operator)

kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio conf get alluxio.cluster.name

Docker / Bare-Metal

bin/alluxio conf get alluxio.cluster.name

alx-ns-alluxio-cluster

The value must equal $CL. If it does not, every key path below is wrong.

Check that etcdctl reaches both ends before going further:

etcdctl --endpoints=$SRC endpoint health
etcdctl --endpoints=$DST endpoint health

http://alluxio-cluster-etcd.alx-ns:2379 is healthy: successfully committed proposal: took = 1.9ms
http://new-etcd:2379 is healthy: successfully committed proposal: took = 2.4ms

Step 1: Record a baseline

Both outputs must be unchanged after the cutover.

Kubernetes (Operator)

# Worker identities
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio info nodes

# Mount URIs — compare byte for byte afterwards, including the scheme
kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio mount list

Docker / Bare-Metal

bin/alluxio info nodes
bin/alluxio mount list

Save both outputs. Step 7 compares against them.

WorkerId        Address          Status
worker-1        10.0.1.11:29999  ONLINE
worker-2        10.0.1.12:29999  ONLINE

Step 2: Check the target etcd

The target may already hold unrelated keys — that is fine, Alluxio only reads and writes under its own prefixes. What must not exist is state for this cluster.name.

for P in /alluxio/CONF/$CL/ /LE/license/$CL /LE/version/$CL /DHT/$CL /ServiceDiscovery/$CL; do
  echo -n "$P -> "; etcdctl --endpoints=$DST get "$P" --prefix --keys-only | grep -c .
done

/alluxio/CONF/alx-ns-alluxio-cluster/ -> 0
/LE/license/alx-ns-alluxio-cluster -> 0
/LE/version/alx-ns-alluxio-cluster -> 0
/DHT/alx-ns-alluxio-cluster -> 0
/ServiceDiscovery/alx-ns-alluxio-cluster -> 0

A non-zero count means a previous or aborted install already used this etcd with this cluster.name. Clear those prefixes, or choose a different cluster.name, before copying — otherwise you layer onto stale state.

Step 3: Copy the durable keys

# Idempotent — re-running overwrites the same keys with the same bytes
{ etcdctl --endpoints=$SRC get /alluxio/CONF/$CL/ --prefix --keys-only | grep -v '^$'; \
  echo "/LE/license/$CL"; echo "/LE/version/$CL"; } \
  | while read -r K; do
      [ -z "$K" ] && continue
      etcdctl --endpoints=$SRC get "$K" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/' | base64 -d \
        | etcdctl --endpoints=$DST put "$K" >/dev/null && echo "copied: $K"
    done

copied: /alluxio/CONF/alx-ns-alluxio-cluster/mount/s3a:%2F%2Fbucket
copied: /alluxio/CONF/alx-ns-alluxio-cluster/cachefilter/default
copied: /LE/license/alx-ns-alluxio-cluster
copied: /LE/version/alx-ns-alluxio-cluster

One copied: line per key. A key that prints nothing did not transfer — Step 4 catches it.

Step 4: Verify the copy

Every line must say MATCH.

vfy(){ a=$(etcdctl --endpoints=$SRC get "$1" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/'); \
       b=$(etcdctl --endpoints=$DST get "$1" -w json | sed -E 's/.*"value":"([^"]+)".*/\1/'); \
       [ "$a" = "$b" ] && echo "MATCH    $1" || echo "MISMATCH $1"; }

{ etcdctl --endpoints=$SRC get /alluxio/CONF/$CL/ --prefix --keys-only | grep -v '^$'; \
  echo "/LE/license/$CL"; echo "/LE/version/$CL"; } \
  | while read -r K; do [ -z "$K" ] && continue; vfy "$K"; done

MATCH    /alluxio/CONF/alx-ns-alluxio-cluster/mount/s3a:%2F%2Fbucket
MATCH    /alluxio/CONF/alx-ns-alluxio-cluster/cachefilter/default
MATCH    /LE/license/alx-ns-alluxio-cluster
MATCH    /LE/version/alx-ns-alluxio-cluster

A MISMATCH means that key did not survive the round-trip. Re-copy it before going on. Do not continue with any line reading MISMATCH.

Step 5: Dump the source for rollback

etcdctl --endpoints=$SRC get "" --prefix -w json > source-keyspace-full.json

Keep this. It is what Rollback falls back to if the old etcd volumes are gone.

Step 6: Repoint and restart

In the AlluxioCluster CR, set etcd.enabled: false and add alluxio.etcd.endpoints. Do not change metadata.name, the mount, or alluxio.cluster.name — the cluster name is what every copied key is prefixed with.

# Idempotent. The Operator deletes the bundled etcd but keeps its PVCs,
# then restarts the coordinator and workers.
kubectl apply -f alluxio-cluster.yaml -n alx-ns

# Wait rather than watch
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=coordinator --timeout=300s
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=worker --timeout=300s

pod/alluxio-cluster-coordinator-0 condition met
pod/alluxio-cluster-worker-0 condition met
pod/alluxio-cluster-worker-1 condition met

FUSE pods must roll too. DaemonSet FUSE pods restart with the cluster; CSI-provisioned ones do not, so restart them and then the application pods that mount them.

kubectl get pods -n alx-ns -l 'app.kubernetes.io/component in (fuse, csi-fuse)'
# Expected: every pod Running, AGE younger than the apply above

On Docker or bare metal there is no CR: set alluxio.etcd.endpoints in alluxio-site.properties on every node, then restart the coordinator, the workers, and the FUSE processes.

Step 7: Verify

Kubernetes (Operator)

kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio info nodes
# Expected: same WorkerIds as Step 1. Addresses may differ; identities may not.

kubectl exec -n alx-ns alluxio-cluster-coordinator-0 -c alluxio-coordinator -- \
  alluxio mount list
# Expected: byte-identical to Step 1, scheme included

Docker / Bare-Metal

bin/alluxio info nodes
bin/alluxio mount list

Then confirm the cache survived. Read the two counters off a worker, re-read a dataset that was cached before the migration, and read them again — see Monitoring for the endpoints:

kubectl -n alx-ns exec alluxio-cluster-worker-0 -- \
  curl -s http://localhost:30000/metrics/ \
  | grep -E 'alluxio_cache_(hit|miss)_calls_total'
# Expected after the re-read: the hit count rises by roughly one per file
# and the miss count does not move. Any rise in misses means that data is
# being fetched from the UFS again — the cache did not survive.

If ring versioning is enabled

Ring member lists do not rebuild themselves, so they need copying too. This only applies when alluxio.user.consistent.hash.ring.versioning.enabled is true; it defaults to false, in which case skip this.

# Empty output means versioning is off — skip
etcdctl --endpoints=$SRC get /DHT/$CL/currentMembers

# Otherwise copy all three lists
for M in currentMembers stagingMembers historicMembers; do
  etcdctl --endpoints=$SRC get /DHT/$CL/$M -w json | sed -E 's/.*"value":"([^"]+)".*/\1/' | base64 -d \
    | etcdctl --endpoints=$DST put /DHT/$CL/$M >/dev/null 2>&1 && echo "copied: /DHT/$CL/$M"
done

See Hash Ring and Worker Lifecycle for what these lists do.

Rollback

If the old etcd PVCs still exist — the normal case, since Step 6 keeps them — revert the CR (etcd.enabled: true, drop the alluxio.etcd.endpoints line) and apply. The Operator re-adopts the old PVCs by name.

kubectl apply -f alluxio-cluster.yaml -n alx-ns
kubectl wait --for=condition=Ready pod -n alx-ns \
  -l app.kubernetes.io/component=coordinator --timeout=300s
# Expected: "condition met". Then re-run Step 7 against the old etcd.

If they are already deleted, stand up an empty etcd and replay the durable keys from the Step 5 dump into it, then point the CR at that:

python3 - source-keyspace-full.json http://fresh-etcd:2379 <<'PY'
import json, sys, base64, subprocess
d = json.load(open(sys.argv[1])); dst = sys.argv[2]
KEEP = ("/alluxio/CONF/", "/LE/license/", "/LE/version/")
for kv in d["kvs"]:
    if int(kv.get("lease", 0) or 0):
        continue                      # leased keys rebuild themselves
    k = base64.b64decode(kv["key"]).decode()
    if not k.startswith(KEEP):
        continue
    v = base64.b64decode(kv.get("value", "") or "")
    subprocess.run(["etcdctl", "--endpoints", dst, "put", k], input=v, check=True)
    print("restored", k)
PY

Cleanup

Only after Step 7 passes. Deleting these volumes is the point of no return for the simple rollback above.

kubectl get pvc -n alx-ns | grep etcd
kubectl delete pvc data-alluxio-cluster-etcd-0 data-alluxio-cluster-etcd-1 \
  data-alluxio-cluster-etcd-2 -n alx-ns

Rebuilding etcd

Alluxio tolerates an etcd outage for a grace period (typically 24 hours) without disrupting I/O, so an etcd pod that restarts cleanly needs no action. If only one member is stuck while the rest of the cluster is healthy, see Recovering a Single etcd Member instead — it is much less work. Rebuild only when the cluster is unrecoverable and its data cannot be copied out — otherwise migrate, which keeps your state.

Rebuilding discards cluster state, including the mount table. Expect to re-create mounts afterwards.

1. Shut down the Alluxio cluster: kubectl delete -f alluxio-cluster.yaml

2. Delete the etcd PVCs: kubectl -n alx-ns delete pvc -l app.kubernetes.io/component=etcd

3. Check the storage class the PVCs used (kubectl -n alx-ns get pvc -l app.kubernetes.io/component=etcd -o jsonpath='{.items[*].spec.storageClassName}' — run this before step 2 if you have not deleted the PVCs yet). If it provisions dynamically, deleting the PVC already reclaimed the volume — skip to step 4. If it is a statically-provisioned local PV, log into each Kubernetes node that hosted an etcd pod and delete the contents of the host path directory used by the etcd PV.

4. Recreate the cluster: kubectl create -f alluxio-cluster.yaml. The Operator provisions a new, empty etcd cluster.

5. Re-mount your UFS paths. If you manage mounts with the UnderFileSystem CRD they are restored for you; otherwise re-add them with alluxio mount add --path <path> --ufs-uri <uri>.

A three-replica etcd cluster only survives a node loss if the replicas are on distinct nodes — see Preventing repeat quorum loss.

Related

• Prerequisites — etcd — node count and quorum rules before you install

• Multiple AZ — independent, shared, and external etcd topologies across zones

• TLS — securing the etcd connection

• Monitoring — etcd availability alerting

• Job Service — a separate etcd for job scheduling

• Installing on Kubernetes — etcd symptoms seen during installation