> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/administration/troubleshooting-alluxio.md). # Troubleshooting This guide helps you diagnose an Alluxio cluster running on Kubernetes. It moves from the symptom you observed, to the health check or log that confirms the cause, to the recovery procedure. If you are running Alluxio outside Kubernetes, the log contents and recovery steps still apply, but the `kubectl` commands do not. ## Before You Start You need `kubectl` access to two namespaces, and it matters which one you are looking in: | Namespace | In the examples | What runs there | | -------------------- | ------------------ | --------------------------------------- | | Your Alluxio cluster | `alx-ns` | Coordinator, workers, FUSE pods, etcd | | The Alluxio Operator | `alluxio-operator` | Operator, CSI driver, doctor controller | Almost every "the command returned nothing" problem is one of these two being swapped. Find yours: ```shell # The namespace holding your Alluxio cluster kubectl get alluxiocluster -A # The namespace holding the Operator, CSI driver, and doctor controller kubectl get pod -A -l app.kubernetes.io/component=doctor-controller ``` Substitute your own namespaces for `alx-ns` and `alluxio-operator` throughout this page. ## 1. Find Your Symptom Start from what you actually saw. Each row links to the section that diagnoses it. | What you observed | Most likely area | Go to | | --------------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------- | | Application reports `Transport endpoint is not connected` | The FUSE mount was lost | [Errors your application sees](#errors-your-application-sees) | | Application reports `Input/output error` on a cached path | Worker or UFS failure behind the mount | [Errors your application sees](#errors-your-application-sees) | | Application hangs on file access, no error | Worker unreachable, or UFS timing out | [Check cluster health](#id-2.-check-cluster-health) | | A pod is not `READY`, or keeps restarting | Component failure | [Check cluster health](#id-2.-check-cluster-health) | | Reads are slower than expected, cache hit rate dropped | Cache or worker health | [What the dashboards say](#what-the-dashboards-say) | | Writes fail after the cache filled up | Page store capacity | [Worker failures](#worker-failures) | | A worker takes minutes to become `READY` after a restart | Page store restore | [Worker failures](#worker-failures) | | S3 API returns 404 `NoSuchBucket` for a path that exists | Path is not mounted | [S3 API errors](#s3-api-errors) | | S3 API endpoint refuses connections | S3 API not enabled | [S3 API errors](#s3-api-errors) | | `alluxio fs` commands fail or hang | Coordinator or etcd | [Coordinator failures](#coordinator-failures) | | Alluxio support asked you for a diagnostic bundle | — | [Collecting a snapshot with Doctor](#id-4.-collecting-a-snapshot-with-doctor) | If your symptom is not listed, work through sections 2 and 3 in order, then collect a snapshot for support. ## 2. Check Cluster Health These checks take under a minute and rule out the majority of causes. ### Are all the pods ready? A `Running` status is not enough — the `READY` column must show every container in the pod is healthy. Coordinator: ```shell kubectl -n alx-ns get pod -l app.kubernetes.io/component=coordinator ``` Workers: ```shell kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker ``` ```console NAME READY STATUS RESTARTS AGE alluxio-cluster-worker-59476bf8c5-lg4sc 1/1 Running 0 46h alluxio-cluster-worker-59476bf8c5-vg6lc 1/1 Running 0 46h ``` FUSE pods, both the DaemonSet and CSI flavors: ```shell kubectl -n alx-ns get pod -l 'app.kubernetes.io/component in (fuse, csi-fuse)' ``` ```console NAME READY STATUS RESTARTS AGE alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0 1/1 Running 0 57m ``` The integrated etcd cluster: ```shell kubectl -n alx-ns get pod -l 'app.kubernetes.io/component=etcd,app.kubernetes.io/instance=alluxio-cluster' ``` ```console NAME READY STATUS RESTARTS AGE alluxio-cluster-etcd-0 1/1 Running 0 46h alluxio-cluster-etcd-1 1/1 Running 0 46h alluxio-cluster-etcd-2 1/1 Running 0 46h ``` Read the `RESTARTS` column as carefully as `READY`. A pod that is ready now but has restarted several times points at a recurring failure — see [Common failures](#id-5.-common-failures-and-recovery) and check the previous container's log. For a quick readiness ratio across a component: ```shell kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker -o jsonpath='{range .items[*]}{.status.containerStatuses[0].ready}{"\n"}{end}' | awk 'BEGIN{t=0}{s+=1;if($1=="true")t+=1}END{print t,"ready /",s,"expected =",t/s*100,"%"}' ``` ```console 2 ready / 2 expected = 100 % ``` ### Can Alluxio reach your storage? If the pods are healthy but reads fail, the under file system (UFS) is the next suspect. Run these from inside a worker or coordinator pod, where the Alluxio CLI and the cluster configuration are available: ```shell kubectl -n alx-ns exec -it deploy/alluxio-cluster-coordinator -- bash ``` Check basic UFS operations: ```shell ./bin/alluxio exec ufsTest --path s3://your_bucket/test_path ``` ```console Running test: createAtomicTest... Passed the test! time: 5205ms ... Tests completed with 0 failed. ``` Check UFS read/write throughput — this example writes and reads a 512MB file with two threads: ```shell ./bin/alluxio exec ufsIOTest --path s3://test_bucket/test_path --io-size 512m --threads 2 ``` ```console { "readSpeedStat" : { ... }, "writeSpeedStat" : { ... }, "errors" : [ ], ... } ``` Zero failures means the UFS is reachable and the credentials, region, and endpoint are configured correctly. Failures here mean the problem is between Alluxio and your storage, not inside Alluxio. ### What the dashboards say The Grafana dashboard is the fastest way to see whether a problem is cluster-wide or confined to one component. Three panels answer most questions: | Metric | Labels | What a problem looks like | | --------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | `alluxio_data_access_bytes_count` | `method` | A sudden drop in `irate(...[5m])` means clients stopped reaching workers. A spike means an unexpected workload change. | | `alluxio_ufs_error` | `ufs_type`, `error_code` | Any sustained increase. Group by `error_code` — it distinguishes a permissions problem from a connectivity one. | | `alluxio_ufs_data_access` | `method` | UFS traffic rising while client traffic is flat means cache misses went up — data is being re-fetched that used to be cached. | | `alluxio_fuse_result` | `method`, `state` | Failures grouped by `method` show which POSIX operation is failing. | A sudden drop in cache hit rate usually means either workers became unhealthy and their cache was lost, or the access pattern changed. The pod checks above distinguish the two. See [Metrics](/ee-ai-en/ai-3.8-15.1.x/reference/metrics.md) for the full list. ## 3. Read the Logs ### Which log answers which question Alluxio spreads one request across several components, so the log that shows the *cause* is often not the one that showed the *error*. Pick by what you need to know: | Component | Namespace | Selector | What you will find there | | -------------- | --------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Coordinator | cluster | `app.kubernetes.io/component=coordinator` | The job service: scheduling of asynchronous `load` and `free` jobs, and their history | | Worker | cluster | `app.kubernetes.io/component=worker` | Serving reads and writes, page store (cache) operations, UFS fetches, cache eviction, page store restore on startup | | FUSE | cluster | `app.kubernetes.io/component in (fuse, csi-fuse)` | POSIX operations from your application, and the errors it sees — this is where an application-visible failure surfaces first | | CSI nodeplugin | operator | `app.kubernetes.io/component=csi-nodeplugin` | Mounting and unmounting the volume on one node — check when a pod cannot start because its volume will not mount | | CSI controller | operator | `app.kubernetes.io/component=csi-controller` | Creation and deletion of per-volume FUSE pods | | etcd | cluster | `app.kubernetes.io/component=etcd` | Quorum and membership problems | Working rule: start at the component your application talked to (FUSE for a mounted path, a worker for the S3 API), then follow the error upstream. ### Reading a component log You rarely know the pod name up front, so select by component instead. The coordinator is a single pod: ```shell kubectl -n alx-ns logs -l app.kubernetes.io/component=coordinator --tail=-1 ``` Workers are many, so prefix each line with the pod it came from: ```shell kubectl -n alx-ns logs -l app.kubernetes.io/component=worker --prefix --tail=-1 ``` {% hint style="warning" %} `kubectl logs -l` returns only the last 10 lines per pod unless you pass `--tail=-1`. Without it you will silently miss the error you are looking for. {% endhint %} For one specific pod, name it directly: ```shell kubectl -n alx-ns logs alluxio-cluster-worker-59476bf8c5-lg4sc ``` Narrow to problems, showing the line after each match so stack traces stay readable: ```shell kubectl -n alx-ns logs alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0 | grep -A 1 'WARN\|ERROR' ``` ```console 2024-07-04 17:29:53,499 ERROR HdfsUfsStatusIterator - Failed to list the path hdfs://localhost:9000/ java.net.ConnectException: Call From myhost/192.168.1.10 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused ``` If the pod has restarted, the interesting log belongs to the *previous* container — the current one started clean: ```shell kubectl -n alx-ns logs -p alluxio-cluster-worker-59476bf8c5-lg4sc ``` `kubectl logs` only shows what the current container wrote to stdout. Alluxio also writes rotated log files inside the pod, under `/opt/alluxio/logs` by default (`alluxio.logs.dir`), which reach further back: ```shell # What is available kubectl -n alx-ns exec alluxio-cluster-worker-59476bf8c5-lg4sc -- ls -lht /opt/alluxio/logs # Copy the current worker log out; coordinator pods have coordinator.log kubectl -n alx-ns cp alluxio-cluster-worker-59476bf8c5-lg4sc:/opt/alluxio/logs/worker.log ./worker.log ``` Rotated files are named `worker--.log`. A [Doctor snapshot](#id-4.-collecting-a-snapshot-with-doctor) gathers all of these from every component in one step, which is usually easier than copying them pod by pod. {% hint style="info" %} Container logs are lost when a pod is deleted and recreated. If you are chasing an intermittent failure, collect a [snapshot with Doctor](#id-4.-collecting-a-snapshot-with-doctor) while the evidence is still there. {% endhint %} ### Reading the CSI driver log When a FUSE volume will not mount, the answer is in the CSI node plugin running on the *same node* as your application pod: ```shell # 1. Find the node your application or FUSE pod is running on PODNS=alx-ns POD=alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0 NODE_NAME=$(kubectl get pod -o jsonpath='{.spec.nodeName}' -n ${PODNS} ${POD}) # 2. Find the CSI node plugin pod on that node CSI_POD_NAME=$(kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=csi-nodeplugin --field-selector spec.nodeName=${NODE_NAME} -o jsonpath='{..metadata.name}') # 3. Read its log kubectl -n alluxio-operator logs -c csi-nodeserver ${CSI_POD_NAME} ``` ## 4. Collecting a Snapshot with Doctor Doctor is the Operator's diagnostic collector. It bundles configuration, logs, metrics, hardware details, and cluster state into a single archive. Collect one when you have exhausted the checks above, or when Alluxio support asks for it. You drive Doctor by creating a `CollectInfo` resource; the doctor controller watches for it and does the work. ### Prerequisites The doctor controller must be running in the operator namespace. If it is missing, upgrade the Alluxio Operator. ```shell kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=doctor-controller ``` ```console NAME READY STATUS RESTARTS AGE alluxio-doctor-controller-cc49c56b6-wlw8k 1/1 Running 0 19s ``` ### Collecting the snapshot Every cluster the Operator creates already includes a `CollectInfo` that runs daily at midnight UTC, collecting everything from the past 24 hours and keeping each archive for 180 days. List what your cluster has: ```shell kubectl -n alx-ns get collectinfo ``` ```console NAME LASTSCHEDULETIME LASTSUCCESSFULTIME AGE alluxio-cluster 2026-09-03T00:00:00Z 2026-09-03T00:04:12Z 46h ``` To capture the state *now* rather than waiting for the next run, apply a one-time `CollectInfo`: ```shell kubectl apply -f - <<'EOF' apiVersion: k8s-operator.alluxio.com/v1 kind: CollectInfo metadata: name: one-time-snapshot # Must be the namespace your Alluxio cluster runs in namespace: alx-ns spec: # Both fields below are already the defaults and can be omitted. What makes # this a one-time collection is the absence of spec.scheduled.cron. type: - all logs: sinceSeconds: 86400 EOF ``` Keep it in a file instead when you are defining a recurring collection — that one belongs in version control alongside your cluster manifests. {% hint style="info" %} Older examples trigger a single run with `spec.scheduled.enabled: false`. That field is deprecated and is not what the controller checks — a `CollectInfo` runs once whenever `spec.scheduled.cron` is absent, and on a schedule whenever it is present. {% endhint %} Confirm it finished — `LASTSUCCESSFULTIME` is populated once the archive is written: ```shell kubectl -n alx-ns get collectinfo one-time-snapshot ``` This collects everything from the past day, which is the right default for most incidents — see [Changing what is collected](#changing-what-is-collected) if it is not. ### Finding and downloading the snapshot Each run writes one `.tar.gz` archive to `/data/doctor` inside the doctor-controller pod, named after the `CollectInfo` that produced it: ``` __.tar.gz ``` For the `one-time-snapshot` above, that is `one-time-snapshot_alx-ns_2026-09-03-07-29-01.tar.gz`. ```shell # 1. Get the doctor controller pod name DOCTOR_NAME=$(kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=doctor-controller -o jsonpath="{.items[0].metadata.name}") # 2. List the archives this CollectInfo produced, newest first kubectl -n alluxio-operator exec ${DOCTOR_NAME} -- ls -lht /data/doctor/one-time-snapshot_alx-ns_*.tar.gz # 3. Copy one archive to your local machine ARCHIVE_NAME=one-time-snapshot_alx-ns_2026-09-03-07-29-01.tar.gz kubectl -n alluxio-operator cp ${DOCTOR_NAME}:/data/doctor/${ARCHIVE_NAME} ./${ARCHIVE_NAME} ``` Drop the filename filter (`ls -lht /data/doctor`) to see every archive, or copy the whole directory (`kubectl -n alluxio-operator cp ${DOCTOR_NAME}:/data/doctor ./doctor`) when you need all of them. A cluster on the default daily schedule accumulates one archive per day, so the directory copy can be large. ### Sending the snapshot to Alluxio By default archives stay in your cluster and you send them to support yourself. Alluxio can also provide credentials for a bucket it maintains, so each snapshot uploads automatically — add an `upload` block to the `CollectInfo`. See [`spec.upload`](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/administration/pages/CEBKKfzsimf31nh9kC4W#spec.upload). ### Changing what is collected The defaults suit most incidents. The two adjustments that come up most often: | You need | Add to `spec` | | ----------------------------------- | ------------------------------------------------------------------------------------ | | Logs from further back than one day | `logs: {sinceSeconds: 259200}` for three days | | A smaller, faster archive | `type: [config, dynamic-config, logs, meta]` — omits `metrics`, the largest category | Every field, including the collection schedule, retention, and metrics resolution, is documented in the [CollectInfo Reference](/ee-ai-en/ai-3.8-15.1.x/reference/collectinfo-crd.md). ## 5. Common Failures and Recovery ### Errors your application sees These surface in your application, not in Alluxio. The FUSE log is where you confirm the cause. | Error | What happened | What to do | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `Transport endpoint is not connected` | The FUSE process serving that mount went away. The mount point still exists in your application's namespace but nothing is behind it, and it does not heal on its own. | Confirm the FUSE pod is `READY` again, then restart the application pod to pick up the new mount. | | `Input/output error` | A read or write reached Alluxio and failed — usually the worker holding the data is unreachable, or the UFS returned an error. | Check the FUSE log for the underlying exception, then follow it to the worker or UFS. | | Operations hang with no error | The worker or the UFS is not responding, and the request is waiting on a timeout. | Check worker readiness, then run `ufsTest` from a coordinator pod. | A FUSE pod stuck in `Init` never mounted at all — it is waiting on a dependency, usually etcd not yet being ready. ### Worker failures Alluxio tolerates worker loss by design: Kubernetes restarts the pod, the cached data on it is gone, and reads fall back to the UFS. I/O does not fail, but it gets slower until the cache refills. | Symptom | Cause | What to do | | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Worker log: `Insufficient page store space` | The page store is full and cannot accept new pages. | Increase the page store size, or check that eviction is keeping up. See [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md). | | Worker fails to start: `quota (...) exceeds the total disk space (...) on ...` | The configured page store size plus reserved size is larger than the volume backing it. | Lower the page store size, or give the volume more capacity. | | Worker takes minutes to become `READY` after a restart, log shows `Page store finished restoring N pages in M ms` | The page store is rebuilt sequentially on startup. A large cache genuinely takes minutes. | Nothing is wrong. Expect longer worker restarts as the cache grows, and plan rolling restarts accordingly. | | Worker pod `OOMKilled`, or evicted by the kubelet | Memory limit too low for the configured heap and direct memory, or an `emptyDir` page store outgrew its `sizeLimit`. Kubelet eviction is not Alluxio eviction. | `kubectl -n alx-ns describe pod ` shows which. Raise the limit, or move the page store to a volume with real capacity. | | Log mentions clocks `being out of sync` between client and worker | Clock skew between nodes invalidates cache freshness checks and forces extra work. | Check NTP on the Kubernetes nodes. | ### Coordinator failures The coordinator runs the job service, which manages asynchronous jobs such as distributed loads. It persists job history and recovers on restart, and Kubernetes restarts a failed coordinator pod automatically. If the job history is corrupted, unfinished jobs are lost and must be resubmitted. Client commands such as `alluxio fs` that hang rather than fail usually point at the coordinator or at etcd behind it. ### FUSE failures A crashed or unresponsive FUSE pod is restarted by its controller — the DaemonSet or the CSI driver. To force a restart of a hung pod: ```shell kubectl -n alx-ns delete pod ``` Applications holding a mount from the old pod see `Transport endpoint is not connected` until they are restarted. ### etcd failures Alluxio tolerates an etcd outage for a grace period (typically 24 hours) without disrupting I/O, so an etcd pod that restarts cleanly needs no action. If the cluster is unrecoverable it has to be rebuilt, which discards the mount table — see [Rebuilding etcd](/ee-ai-en/ai-3.8-15.1.x/administration/managing-etcd.md#rebuilding-etcd). If etcd is healthy and you only want to move to a different one, migrate instead and keep your state: [Migrating to a Different etcd](/ee-ai-en/ai-3.8-15.1.x/administration/managing-etcd.md#migrating-to-a-different-etcd). ### S3 API errors The S3 API is served by the workers themselves, on the port set by `alluxio.worker.rest.port` (default `29998`). There is no separate proxy pod. | Symptom | Cause | What to do | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Connection refused on the S3 endpoint | The S3 API is not enabled. | Set `alluxio.worker.s3.api.enabled=true`. The Alluxio 2.x property `alluxio.proxy.s3.enabled` does not exist in 3.x and has no effect. | | `404 NoSuchBucket` for a path you expect to exist | The first path segment after the endpoint is the Alluxio mount name, and it did not resolve to a mount. | List the mount table and confirm the name matches. | | `Timeout waiting for connection from pool` in worker logs | The S3 client's connection pool is exhausted, typically under high concurrency or after connections leaked from an earlier run. | Set `alluxio.underfs.s3.connections.max` explicitly for high-concurrency workloads. | ## Related * [CollectInfo Reference](/ee-ai-en/ai-3.8-15.1.x/reference/collectinfo-crd.md) — every field of the diagnostic collection resource * [Metrics](/ee-ai-en/ai-3.8-15.1.x/reference/metrics.md) — the full metric list behind the dashboards * [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md) — managing page store capacity * [Installing on Kubernetes](/ee-ai-en/ai-3.8-15.1.x/start/installing-on-kubernetes.md) — the Operator and CSI components referenced here