Troubleshooting
This guide helps you diagnose an Alluxio cluster running on Kubernetes. It moves from the symptom you observed, to the health check or log that confirms the cause, to the recovery procedure.
If you are running Alluxio outside Kubernetes, the log contents and recovery steps still apply, but the kubectl commands do not.
Before You Start
You need kubectl access to two namespaces, and it matters which one you are looking in:
Your Alluxio cluster
alx-ns
Coordinator, workers, FUSE pods, etcd
The Alluxio Operator
alluxio-operator
Operator, CSI driver, doctor controller
Almost every "the command returned nothing" problem is one of these two being swapped. Find yours:
# The namespace holding your Alluxio cluster
kubectl get alluxiocluster -A
# The namespace holding the Operator, CSI driver, and doctor controller
kubectl get pod -A -l app.kubernetes.io/component=doctor-controllerSubstitute your own namespaces for alx-ns and alluxio-operator throughout this page.
1. Find Your Symptom
Start from what you actually saw. Each row links to the section that diagnoses it.
Application reports Transport endpoint is not connected
The FUSE mount was lost
Application reports Input/output error on a cached path
Worker or UFS failure behind the mount
If your symptom is not listed, work through sections 2 and 3 in order, then collect a snapshot for support.
2. Check Cluster Health
These checks take under a minute and rule out the majority of causes.
Are all the pods ready?
A Running status is not enough — the READY column must show every container in the pod is healthy.
Coordinator:
Workers:
FUSE pods, both the DaemonSet and CSI flavors:
The integrated etcd cluster:
Read the RESTARTS column as carefully as READY. A pod that is ready now but has restarted several times points at a recurring failure — see Common failures and check the previous container's log.
For a quick readiness ratio across a component:
Can Alluxio reach your storage?
If the pods are healthy but reads fail, the under file system (UFS) is the next suspect. Run these from inside a worker or coordinator pod, where the Alluxio CLI and the cluster configuration are available:
Check basic UFS operations:
Check UFS read/write throughput — this example writes and reads a 512MB file with two threads:
Zero failures means the UFS is reachable and the credentials, region, and endpoint are configured correctly. Failures here mean the problem is between Alluxio and your storage, not inside Alluxio.
What the dashboards say
The Grafana dashboard is the fastest way to see whether a problem is cluster-wide or confined to one component. Three panels answer most questions:
alluxio_data_access_bytes_count
method
A sudden drop in irate(...[5m]) means clients stopped reaching workers. A spike means an unexpected workload change.
alluxio_ufs_error
ufs_type, error_code
Any sustained increase. Group by error_code — it distinguishes a permissions problem from a connectivity one.
alluxio_ufs_data_access
method
UFS traffic rising while client traffic is flat means cache misses went up — data is being re-fetched that used to be cached.
alluxio_fuse_result
method, state
Failures grouped by method show which POSIX operation is failing.
A sudden drop in cache hit rate usually means either workers became unhealthy and their cache was lost, or the access pattern changed. The pod checks above distinguish the two.
See Metrics for the full list.
3. Read the Logs
Which log answers which question
Alluxio spreads one request across several components, so the log that shows the cause is often not the one that showed the error. Pick by what you need to know:
Coordinator
cluster
app.kubernetes.io/component=coordinator
The job service: scheduling of asynchronous load and free jobs, and their history
Worker
cluster
app.kubernetes.io/component=worker
Serving reads and writes, page store (cache) operations, UFS fetches, cache eviction, page store restore on startup
FUSE
cluster
app.kubernetes.io/component in (fuse, csi-fuse)
POSIX operations from your application, and the errors it sees — this is where an application-visible failure surfaces first
CSI nodeplugin
operator
app.kubernetes.io/component=csi-nodeplugin
Mounting and unmounting the volume on one node — check when a pod cannot start because its volume will not mount
CSI controller
operator
app.kubernetes.io/component=csi-controller
Creation and deletion of per-volume FUSE pods
etcd
cluster
app.kubernetes.io/component=etcd
Quorum and membership problems
Working rule: start at the component your application talked to (FUSE for a mounted path, a worker for the S3 API), then follow the error upstream.
Reading a component log
You rarely know the pod name up front, so select by component instead. The coordinator is a single pod:
Workers are many, so prefix each line with the pod it came from:
For one specific pod, name it directly:
Narrow to problems, showing the line after each match so stack traces stay readable:
If the pod has restarted, the interesting log belongs to the previous container — the current one started clean:
kubectl logs only shows what the current container wrote to stdout. Alluxio also writes rotated log files inside the pod, under /opt/alluxio/logs by default (alluxio.logs.dir), which reach further back:
Rotated files are named worker-<date>-<n>.log. A Doctor snapshot gathers all of these from every component in one step, which is usually easier than copying them pod by pod.
Reading the CSI driver log
When a FUSE volume will not mount, the answer is in the CSI node plugin running on the same node as your application pod:
4. Collecting a Snapshot with Doctor
Doctor is the Operator's diagnostic collector. It bundles configuration, logs, metrics, hardware details, and cluster state into a single archive. Collect one when you have exhausted the checks above, or when Alluxio support asks for it.
You drive Doctor by creating a CollectInfo resource; the doctor controller watches for it and does the work.
Prerequisites
The doctor controller must be running in the operator namespace. If it is missing, upgrade the Alluxio Operator.
Collecting the snapshot
Every cluster the Operator creates already includes a CollectInfo that runs daily at midnight UTC, collecting everything from the past 24 hours and keeping each archive for 180 days. List what your cluster has:
To capture the state now rather than waiting for the next run, apply a one-time CollectInfo:
Keep it in a file instead when you are defining a recurring collection — that one belongs in version control alongside your cluster manifests.
Confirm it finished — LASTSUCCESSFULTIME is populated once the archive is written:
This collects everything from the past day, which is the right default for most incidents — see Changing what is collected if it is not.
Finding and downloading the snapshot
Each run writes one .tar.gz archive to /data/doctor inside the doctor-controller pod, named after the CollectInfo that produced it:
For the one-time-snapshot above, that is one-time-snapshot_alx-ns_2026-09-03-07-29-01.tar.gz.
Drop the filename filter (ls -lht /data/doctor) to see every archive, or copy the whole directory (kubectl -n alluxio-operator cp ${DOCTOR_NAME}:/data/doctor ./doctor) when you need all of them. A cluster on the default daily schedule accumulates one archive per day, so the directory copy can be large.
Sending the snapshot to Alluxio
By default archives stay in your cluster and you send them to support yourself. Alluxio can also provide credentials for a bucket it maintains, so each snapshot uploads automatically — add an upload block to the CollectInfo. See spec.upload.
Changing what is collected
The defaults suit most incidents. The two adjustments that come up most often:
You need
Add to spec
Logs from further back than one day
logs: {sinceSeconds: 259200} for three days
A smaller, faster archive
type: [config, dynamic-config, logs, meta] — omits metrics, the largest category
Every field, including the collection schedule, retention, and metrics resolution, is documented in the CollectInfo Reference.
5. Common Failures and Recovery
Errors your application sees
These surface in your application, not in Alluxio. The FUSE log is where you confirm the cause.
Transport endpoint is not connected
The FUSE process serving that mount went away. The mount point still exists in your application's namespace but nothing is behind it, and it does not heal on its own.
Confirm the FUSE pod is READY again, then restart the application pod to pick up the new mount.
Input/output error
A read or write reached Alluxio and failed — usually the worker holding the data is unreachable, or the UFS returned an error.
Check the FUSE log for the underlying exception, then follow it to the worker or UFS.
Operations hang with no error
The worker or the UFS is not responding, and the request is waiting on a timeout.
Check worker readiness, then run ufsTest from a coordinator pod.
A FUSE pod stuck in Init never mounted at all — it is waiting on a dependency, usually etcd not yet being ready.
Worker failures
Alluxio tolerates worker loss by design: Kubernetes restarts the pod, the cached data on it is gone, and reads fall back to the UFS. I/O does not fail, but it gets slower until the cache refills.
Worker log: Insufficient page store space
The page store is full and cannot accept new pages.
Increase the page store size, or check that eviction is keeping up. See Cache Eviction.
Worker fails to start: quota (...) exceeds the total disk space (...) on ...
The configured page store size plus reserved size is larger than the volume backing it.
Lower the page store size, or give the volume more capacity.
Worker takes minutes to become READY after a restart, log shows Page store finished restoring N pages in M ms
The page store is rebuilt sequentially on startup. A large cache genuinely takes minutes.
Nothing is wrong. Expect longer worker restarts as the cache grows, and plan rolling restarts accordingly.
Worker pod OOMKilled, or evicted by the kubelet
Memory limit too low for the configured heap and direct memory, or an emptyDir page store outgrew its sizeLimit. Kubelet eviction is not Alluxio eviction.
kubectl -n alx-ns describe pod <worker-pod> shows which. Raise the limit, or move the page store to a volume with real capacity.
Log mentions clocks being out of sync between client and worker
Clock skew between nodes invalidates cache freshness checks and forces extra work.
Check NTP on the Kubernetes nodes.
Peers log redirect or connect timeouts to a worker and reads fall back to the UFS, while the worker pod stays READY and does not restart
The worker JVM stops for seconds, for GC or at a safepoint.
See Diagnosing Stalls.
Coordinator failures
The coordinator runs the job service, which manages asynchronous jobs such as distributed loads. It persists job history and recovers on restart, and Kubernetes restarts a failed coordinator pod automatically.
If the job history is corrupted, unfinished jobs are lost and must be resubmitted. Client commands such as alluxio fs that hang rather than fail usually point at the coordinator or at etcd behind it.
FUSE failures
A crashed or unresponsive FUSE pod is restarted by its controller — the DaemonSet or the CSI driver. To force a restart of a hung pod:
Applications holding a mount from the old pod see Transport endpoint is not connected until they are restarted.
etcd failures
Alluxio tolerates an etcd outage for a grace period (typically 24 hours) without disrupting I/O, so an etcd pod that restarts cleanly needs no action.
If the cluster is unrecoverable it has to be rebuilt, which discards the mount table — see Rebuilding etcd. If etcd is healthy and you only want to move to a different one, migrate instead and keep your state: Migrating to a Different etcd.
S3 API errors
The S3 API is served by the workers themselves, on the port set by alluxio.worker.rest.port (default 29998). There is no separate proxy pod.
Connection refused on the S3 endpoint
The S3 API is not enabled.
Set alluxio.worker.s3.api.enabled=true. The Alluxio 2.x property alluxio.proxy.s3.enabled does not exist in 3.x and has no effect.
404 NoSuchBucket for a path you expect to exist
The first path segment after the endpoint is the Alluxio mount name, and it did not resolve to a mount.
List the mount table and confirm the name matches.
Timeout waiting for connection from pool in worker logs
The S3 client's connection pool is exhausted, typically under high concurrency or after connections leaked from an earlier run.
Set alluxio.underfs.s3.connections.max explicitly for high-concurrency workloads.
Related
CollectInfo Reference — every field of the diagnostic collection resource
Metrics — the full metric list behind the dashboards
Cache Eviction — managing page store capacity
Installing on Kubernetes — the Operator and CSI components referenced here
Last updated