> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/administration/troubleshooting-alluxio.md).

# Troubleshooting

This guide helps you diagnose an Alluxio cluster running on Kubernetes. It moves from the symptom you observed, to the health check or log that confirms the cause, to the recovery procedure.

If you are running Alluxio outside Kubernetes, the log contents and recovery steps still apply, but the `kubectl` commands do not.

## Before You Start

You need `kubectl` access to two namespaces, and it matters which one you are looking in:

| Namespace            | In the examples    | What runs there                       |
| -------------------- | ------------------ | ------------------------------------- |
| Your Alluxio cluster | `alx-ns`           | Coordinator, workers, FUSE pods, etcd |
| The Alluxio Operator | `alluxio-operator` | Operator, CSI driver                  |

Almost every "the command returned nothing" problem is one of these two being swapped. Find yours:

```shell
# The namespace holding your Alluxio cluster
kubectl get alluxiocluster -A

# The namespace holding the Operator and CSI driver
kubectl get pod -A -l app.kubernetes.io/name=alluxio-operator
```

Substitute your own namespaces for `alx-ns` and `alluxio-operator` throughout this page.

## 1. Find Your Symptom

Start from what you actually saw. Each row links to the section that diagnoses it.

| What you observed                                                                                       | Most likely area                       | Go to                                                                                                                                            |
| ------------------------------------------------------------------------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Application reports `Transport endpoint is not connected`                                               | The FUSE mount was lost                | [Errors your application sees](#errors-your-application-sees)                                                                                    |
| Application reports `Input/output error` on a cached path                                               | Worker or UFS failure behind the mount | [Errors your application sees](#errors-your-application-sees)                                                                                    |
| Application hangs on file access, no error                                                              | Worker unreachable, or UFS timing out  | [Check cluster health](#id-2.-check-cluster-health)                                                                                              |
| A pod is not `READY`, or keeps restarting                                                               | Component failure                      | [Check cluster health](#id-2.-check-cluster-health)                                                                                              |
| Reads are slower than expected, cache hit rate dropped                                                  | Cache or worker health                 | [What the dashboards say](#what-the-dashboards-say)                                                                                              |
| Writes fail after the cache filled up                                                                   | Page store capacity                    | [Worker failures](#worker-failures)                                                                                                              |
| A worker is smaller than configured, refuses to start with disk errors, or logs `Stopping this worker:` | Page store or metastore disk           | [Disk Health — Recover from a Disk Failure](https://documentation.alluxio.io/ee-ai-en/administration/disk-health.md#recover-from-a-disk-failure) |
| A worker takes minutes to become `READY` after a restart                                                | Page store restore                     | [Worker failures](#worker-failures)                                                                                                              |
| S3 API returns 404 `NoSuchBucket` for a path that exists                                                | Path is not mounted                    | [S3 API errors](#s3-api-errors)                                                                                                                  |
| S3 API endpoint refuses connections                                                                     | S3 API not enabled                     | [S3 API errors](#s3-api-errors)                                                                                                                  |
| `alluxio fs` commands fail or hang                                                                      | Coordinator or etcd                    | [Coordinator failures](#coordinator-failures)                                                                                                    |
| Alluxio support asked you for a diagnostic bundle                                                       | —                                      | [Collecting a diagnostic bundle](#id-4.-collecting-a-diagnostic-bundle)                                                                          |

If your symptom is not listed, work through sections 2 and 3 in order, then collect a diagnostic bundle for support.

## 2. Check Cluster Health

These checks take under a minute and rule out the majority of causes.

### Are all the pods ready?

A `Running` status is not enough — the `READY` column must show every container in the pod is healthy.

Coordinator:

```shell
kubectl -n alx-ns get pod -l app.kubernetes.io/component=coordinator
```

Workers:

```shell
kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker
```

```console
NAME                                      READY   STATUS    RESTARTS   AGE
alluxio-cluster-worker-59476bf8c5-lg4sc   1/1     Running   0          46h
alluxio-cluster-worker-59476bf8c5-vg6lc   1/1     Running   0          46h
```

FUSE pods, both the DaemonSet and CSI flavors:

```shell
kubectl -n alx-ns get pod -l 'app.kubernetes.io/component in (fuse, csi-fuse)'
```

```console
NAME                                           READY   STATUS    RESTARTS   AGE
alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0   1/1     Running   0          57m
```

The integrated etcd cluster:

```shell
kubectl -n alx-ns get pod -l 'app.kubernetes.io/component=etcd,app.kubernetes.io/instance=alluxio-cluster'
```

```console
NAME                     READY   STATUS    RESTARTS   AGE
alluxio-cluster-etcd-0   1/1     Running   0          46h
alluxio-cluster-etcd-1   1/1     Running   0          46h
alluxio-cluster-etcd-2   1/1     Running   0          46h
```

Read the `RESTARTS` column as carefully as `READY`. A pod that is ready now but has restarted several times points at a recurring failure — see [Common failures](#id-5.-common-failures-and-recovery) and check the previous container's log.

For a quick readiness ratio across a component:

```shell
kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker -o jsonpath='{range .items[*]}{.status.containerStatuses[0].ready}{"\n"}{end}' | awk 'BEGIN{t=0}{s+=1;if($1=="true")t+=1}END{print t,"ready /",s,"expected =",t/s*100,"%"}'
```

```console
2 ready / 2 expected = 100 %
```

### Can Alluxio reach your storage?

If the pods are healthy but reads fail, the under file system (UFS) is the next suspect. Run these from inside a worker or coordinator pod, where the Alluxio CLI and the cluster configuration are available:

```shell
kubectl -n alx-ns exec -it deploy/alluxio-cluster-coordinator -- bash
```

Check basic UFS operations:

```shell
./bin/alluxio exec ufsTest --path s3://your_bucket/test_path
```

```console
Running test: createAtomicTest...
Passed the test! time: 5205ms
...
Tests completed with 0 failed.
```

Check UFS read/write throughput — this example writes and reads a 512MB file with two threads:

```shell
./bin/alluxio exec ufsIOTest --path s3://test_bucket/test_path --io-size 512m --threads 2
```

```console
{
  "readSpeedStat" : { ... },
  "writeSpeedStat" : { ... },
  "errors" : [ ],
  ...
}
```

Zero failures means the UFS is reachable and the credentials, region, and endpoint are configured correctly. Failures here mean the problem is between Alluxio and your storage, not inside Alluxio.

### What the dashboards say

The Grafana dashboard is the fastest way to see whether a problem is cluster-wide or confined to one component. Three panels answer most questions:

| Metric                            | Labels                   | What a problem looks like                                                                                                     |
| --------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| `alluxio_data_access_bytes_count` | `method`                 | A sudden drop in `irate(...[5m])` means clients stopped reaching workers. A spike means an unexpected workload change.        |
| `alluxio_ufs_error`               | `ufs_type`, `error_code` | Any sustained increase. Group by `error_code` — it distinguishes a permissions problem from a connectivity one.               |
| `alluxio_ufs_data_access`         | `method`                 | UFS traffic rising while client traffic is flat means cache misses went up — data is being re-fetched that used to be cached. |
| `alluxio_fuse_result`             | `method`, `state`        | Failures grouped by `method` show which POSIX operation is failing.                                                           |

A sudden drop in cache hit rate usually means either workers became unhealthy and their cache was lost, or the access pattern changed. The pod checks above distinguish the two.

See [Metrics](https://documentation.alluxio.io/ee-ai-en/reference/metrics.md) for the full list.

## 3. Read the Logs

### Which log answers which question

Alluxio spreads one request across several components, so the log that shows the *cause* is often not the one that showed the *error*. Pick by what you need to know:

| Component      | Namespace | Selector                                          | What you will find there                                                                                                     |
| -------------- | --------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| Coordinator    | cluster   | `app.kubernetes.io/component=coordinator`         | The job service: scheduling of asynchronous `load` and `free` jobs, and their history                                        |
| Worker         | cluster   | `app.kubernetes.io/component=worker`              | Serving reads and writes, page store (cache) operations, UFS fetches, cache eviction, page store restore on startup          |
| FUSE           | cluster   | `app.kubernetes.io/component in (fuse, csi-fuse)` | POSIX operations from your application, and the errors it sees — this is where an application-visible failure surfaces first |
| CSI nodeplugin | operator  | `app.kubernetes.io/component=csi-nodeplugin`      | Mounting and unmounting the volume on one node — check when a pod cannot start because its volume will not mount             |
| CSI controller | operator  | `app.kubernetes.io/component=csi-controller`      | Creation and deletion of per-volume FUSE pods                                                                                |
| etcd           | cluster   | `app.kubernetes.io/component=etcd`                | Quorum and membership problems                                                                                               |

Working rule: start at the component your application talked to (FUSE for a mounted path, a worker for the S3 API), then follow the error upstream.

### Reading a component log

You rarely know the pod name up front, so select by component instead. The coordinator is a single pod:

```shell
kubectl -n alx-ns logs -l app.kubernetes.io/component=coordinator --tail=-1
```

Workers are many, so prefix each line with the pod it came from:

```shell
kubectl -n alx-ns logs -l app.kubernetes.io/component=worker --prefix --tail=-1
```

{% hint style="warning" %}
`kubectl logs -l` returns only the last 10 lines per pod unless you pass `--tail=-1`. Without it you will silently miss the error you are looking for.
{% endhint %}

For one specific pod, name it directly:

```shell
kubectl -n alx-ns logs alluxio-cluster-worker-59476bf8c5-lg4sc
```

Narrow to problems, showing the line after each match so stack traces stay readable:

```shell
kubectl -n alx-ns logs alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0 | grep -A 1 'WARN\|ERROR'
```

```console
2024-07-04 17:29:53,499 ERROR HdfsUfsStatusIterator - Failed to list the path hdfs://localhost:9000/
java.net.ConnectException: Call From myhost/192.168.1.10 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
```

If the pod has restarted, the interesting log belongs to the *previous* container — the current one started clean:

```shell
kubectl -n alx-ns logs -p alluxio-cluster-worker-59476bf8c5-lg4sc
```

`kubectl logs` only shows what the current container wrote to stdout. Alluxio also writes rotated log files inside the pod, under `/opt/alluxio/logs` by default (`alluxio.logs.dir`), which reach further back:

```shell
# What is available
kubectl -n alx-ns exec alluxio-cluster-worker-59476bf8c5-lg4sc -- ls -lht /opt/alluxio/logs

# Copy the current worker log out; coordinator pods have coordinator.log
kubectl -n alx-ns cp alluxio-cluster-worker-59476bf8c5-lg4sc:/opt/alluxio/logs/worker.log ./worker.log
```

Rotated files are named `worker-<date>-<n>.log`. A [diagnostic bundle](#id-4.-collecting-a-diagnostic-bundle) gathers all of these from every component in one step, which is usually easier than copying them pod by pod.

{% hint style="info" %}
Container logs are lost when a pod is deleted and recreated. If you are chasing an intermittent failure, collect a [diagnostic bundle](#id-4.-collecting-a-diagnostic-bundle) while the evidence is still there.
{% endhint %}

### Reading the CSI driver log

When a FUSE volume will not mount, the answer is in the CSI node plugin running on the *same node* as your application pod:

```shell
# 1. Find the node your application or FUSE pod is running on
PODNS=alx-ns POD=alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0
NODE_NAME=$(kubectl get pod -o jsonpath='{.spec.nodeName}' -n ${PODNS} ${POD})

# 2. Find the CSI node plugin pod on that node
CSI_POD_NAME=$(kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=csi-nodeplugin --field-selector spec.nodeName=${NODE_NAME} -o jsonpath='{..metadata.name}')

# 3. Read its log
kubectl -n alluxio-operator logs -c csi-nodeserver ${CSI_POD_NAME}
```

## 4. Collecting a Diagnostic Bundle

When the checks above do not explain the problem, or Alluxio support asks for it, collect a diagnostic bundle with `alluxioctl doctor bundle`. It runs on your own machine and packs the cluster's configuration, state, jobs, logs, container output, and metric history into one `.tar.gz` file:

```shell
alluxioctl doctor bundle -n alx-ns -c alluxio --description "worker reads hang" -o /data/diag/
```

See [Diagnostic Bundle](https://documentation.alluxio.io/ee-ai-en/administration/diagnostic-bundle.md) for installing `alluxioctl`, the permissions it needs, choosing the time window, thread dumps and JFR data, and what the bundle contains.

## 5. Common Failures and Recovery

### Errors your application sees

These surface in your application, not in Alluxio. The FUSE log is where you confirm the cause.

| Error                                 | What happened                                                                                                                                                          | What to do                                                                                        |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `Transport endpoint is not connected` | The FUSE process serving that mount went away. The mount point still exists in your application's namespace but nothing is behind it, and it does not heal on its own. | Confirm the FUSE pod is `READY` again, then restart the application pod to pick up the new mount. |
| `Input/output error`                  | A read or write reached Alluxio and failed — usually the worker holding the data is unreachable, or the UFS returned an error.                                         | Check the FUSE log for the underlying exception, then follow it to the worker or UFS.             |
| Operations hang with no error         | The worker or the UFS is not responding, and the request is waiting on a timeout.                                                                                      | Check worker readiness, then run `ufsTest` from a coordinator pod.                                |

A FUSE pod stuck in `Init` never mounted at all — it is waiting on a dependency, usually etcd not yet being ready.

### Worker failures

Alluxio tolerates worker loss by design: Kubernetes restarts the pod, the cached data on it is gone, and reads fall back to the UFS. I/O does not fail, but it gets slower until the cache refills.

| Symptom                                                                                                                                    | Cause                                                                                                                                                          | What to do                                                                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Worker log: `Insufficient page store space`                                                                                                | The page store is full and cannot accept new pages.                                                                                                            | Increase the page store size, or check that eviction is keeping up. See [Cache Eviction](https://documentation.alluxio.io/ee-ai-en/cache/cache-eviction.md). |
| Worker fails to start: `quota (...) exceeds the total disk space (...) on ...`                                                             | The configured page store size plus reserved size is larger than the volume backing it.                                                                        | Lower the page store size, or give the volume more capacity.                                                                                                 |
| Worker takes minutes to become `READY` after a restart, log shows `Page store finished restoring N pages in M ms`                          | The page store is rebuilt sequentially on startup. A large cache genuinely takes minutes.                                                                      | Nothing is wrong. Expect longer worker restarts as the cache grows, and plan rolling restarts accordingly.                                                   |
| Worker pod `OOMKilled`, or evicted by the kubelet                                                                                          | Memory limit too low for the configured heap and direct memory, or an `emptyDir` page store outgrew its `sizeLimit`. Kubelet eviction is not Alluxio eviction. | `kubectl -n alx-ns describe pod <worker-pod>` shows which. Raise the limit, or move the page store to a volume with real capacity.                           |
| Log mentions clocks `being out of sync` between client and worker                                                                          | Clock skew between nodes invalidates cache freshness checks and forces extra work.                                                                             | Check NTP on the Kubernetes nodes.                                                                                                                           |
| Peers log redirect or connect timeouts to a worker and reads fall back to the UFS, while the worker pod stays `READY` and does not restart | The worker JVM stops for seconds, for GC or at a safepoint.                                                                                                    | See [Diagnosing Stalls](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md#diagnosing-stalls).                                      |

### Coordinator failures

The coordinator runs the job service, which manages asynchronous jobs such as distributed loads. It persists job history and recovers on restart, and Kubernetes restarts a failed coordinator pod automatically.

If the job history is corrupted, unfinished jobs are lost and must be resubmitted. Client commands such as `alluxio fs` that hang rather than fail usually point at the coordinator or at etcd behind it.

### FUSE failures

A crashed or unresponsive FUSE pod is restarted by its controller — the DaemonSet or the CSI driver. To force a restart of a hung pod:

```shell
kubectl -n alx-ns delete pod <fuse-pod-name>
```

Applications holding a mount from the old pod see `Transport endpoint is not connected` until they are restarted.

### etcd failures

Alluxio tolerates an etcd outage for a grace period (typically 24 hours) without disrupting I/O, so an etcd pod that restarts cleanly needs no action.

If the cluster is unrecoverable it has to be rebuilt, which discards the mount table — see [Rebuilding etcd](https://documentation.alluxio.io/ee-ai-en/administration/managing-etcd.md#rebuilding-etcd). If etcd is healthy and you only want to move to a different one, migrate instead and keep your state: [Migrating to a Different etcd](https://documentation.alluxio.io/ee-ai-en/administration/managing-etcd.md#migrating-to-a-different-etcd).

### S3 API errors

The S3 API is served by the workers themselves, on the port set by `alluxio.worker.rest.port` (default `29998`). There is no separate proxy pod.

| Symptom                                                   | Cause                                                                                                                           | What to do                                                                                                                             |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Connection refused on the S3 endpoint                     | The S3 API is not enabled.                                                                                                      | Set `alluxio.worker.s3.api.enabled=true`. The Alluxio 2.x property `alluxio.proxy.s3.enabled` does not exist in 3.x and has no effect. |
| `404 NoSuchBucket` for a path you expect to exist         | The first path segment after the endpoint is the Alluxio mount name, and it did not resolve to a mount.                         | List the mount table and confirm the name matches.                                                                                     |
| `Timeout waiting for connection from pool` in worker logs | The S3 client's connection pool is exhausted, typically under high concurrency or after connections leaked from an earlier run. | Set `alluxio.underfs.s3.connections.max` explicitly for high-concurrency workloads.                                                    |

## Related

* [Metrics](https://documentation.alluxio.io/ee-ai-en/reference/metrics.md) — the full metric list behind the dashboards
* [Disk Health](https://documentation.alluxio.io/ee-ai-en/administration/disk-health.md) — failing page store and metastore disks
* [Cache Eviction](https://documentation.alluxio.io/ee-ai-en/cache/cache-eviction.md) — managing page store capacity
* [Installing on Kubernetes](https://documentation.alluxio.io/ee-ai-en/start/installing-on-kubernetes.md) — the Operator and CSI components referenced here
