> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-da-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-da-en/administration/troubleshooting-alluxio.md). # Troubleshooting Alluxio This guide provides a structured approach to troubleshooting your Alluxio cluster on Kubernetes. It covers everything from initial health checks to detailed diagnostics and recovery procedures for common issues. ## 1. Initial Health Checks When you encounter an issue, start with these high-level checks to quickly assess the overall health of your Alluxio cluster and its dependencies. ### Checking Component Status Verify that all Alluxio and etcd pods are running and in a `READY` state. A `Running` status is not sufficient; the `READY` column should show that all containers in the pod are healthy. ```shell # Check the readiness of Alluxio coordinator pods $ kubectl -n alx-ns get pod -l app.kubernetes.io/component=coordinator # Check the readiness of Alluxio worker pods $ kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker NAME READY STATUS RESTARTS AGE alluxio-cluster-worker-59476bf8c5-lg4sc 1/1 Running 0 46h alluxio-cluster-worker-59476bf8c5-vg6lc 1/1 Running 0 46h # Check the readiness of the integrated etcd cluster $ kubectl -n alx-ns get pod -l 'app.kubernetes.io/component=etcd,app.kubernetes.io/instance=alluxio-cluster' NAME READY STATUS RESTARTS AGE alluxio-cluster-etcd-0 1/1 Running 0 46h alluxio-cluster-etcd-1 1/1 Running 0 46h alluxio-cluster-etcd-2 1/1 Running 0 46h ``` You can also use this one-liner to get a readiness percentage for a specific component: ```shell # Example for workers: $ kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker -o jsonpath='{range .items[*]}{.status.containerStatuses[0].ready}{"\n"}{end}' | awk 'BEGIN{t=0}{s+=1;if($1=="true")t+=1}END{print t,"ready /",s,"expected =",t/s*100,"%"}' 2 ready / 2 expected = 100 % ``` ### Verifying UFS Connectivity Ensure that Alluxio can communicate with the underlying storage system (UFS). Run the `ufsTest` to check basic UFS operations: ```shell $ ./bin/alluxio exec ufsTest --path s3://your_bucket/test_path Running test: createAtomicTest... Passed the test! time: 5205ms ... Tests completed with 0 failed. ``` Run the `ufsIOTest` to check UFS read/write throughput: ```shell # This example writes and reads a 512MB file with two threads $ ./bin/alluxio exec ufsIOTest --path s3://test_bucket/test_path --io-size 512m --threads 2 { "readSpeedStat" : { ... }, "writeSpeedStat" : { ... }, "errors" : [ ], ... } ``` A successful test with no errors indicates that the UFS is reachable and configured correctly. ### Monitoring Key Metrics via Dashboard The Grafana dashboard provides the quickest way to spot anomalies. Focus on these key areas: * **Liveliness**: Look at the requests-per-second (RPS) for workers (`irate(alluxio_data_access_bytes_count[5m])`). A sudden, unexpected spike or drop can indicate a problem. * **UFS Data Flow**: Monitor the `alluxio_ufs_data_access` and `alluxio_ufs_error` metrics. An increase in errors is a clear sign of UFS connectivity or permission issues. * **Cache Hit Rate**: A sudden drop in the overall cache hit rate can indicate that workers are unhealthy or that the data access pattern has changed unexpectedly. ## 2. Gathering Detailed Diagnostic Information If initial health checks don't reveal the issue, you'll need to dig deeper by inspecting logs and collecting a full diagnostic snapshot. ### Inspecting Logs #### Alluxio Process Logs Check the logs for specific error messages. ```shell # Get all logs from a specific pod (e.g., a worker) $ kubectl -n alx-ns logs alluxio-cluster-worker-59476bf8c5-lg4sc # Filter for WARN or ERROR messages and show the line after the match $ kubectl -n alx-ns logs alluxio-cluster-worker-acee53e8f0a9-3gjbrdekk0 | grep -A 1 'WARN\|ERROR' 2024-07-04 17:29:53,499 ERROR HdfsUfsStatusIterator - Failed to list the path hdfs://localhost:9000/ java.net.ConnectException: Call From myhost/192.168.1.10 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused # Check logs from a previously failed container $ kubectl -n alx-ns logs -p alluxio-cluster-worker-59476bf8c5-lg4sc ``` ### Generating a Diagnostic Snapshot For complex issues, the `collectinfo` tool gathers a comprehensive snapshot of your cluster's state, which is invaluable for offline analysis or for sharing with support. The snapshot includes: * Configuration files * Hardware specifications of Kubernetes nodes * Data from etcd (mounts, quotas, etc.) * Logs from all Alluxio components * Metrics over a specified time range * Job service history #### Prerequisites Ensure the `alluxio-collectinfo-controller` is running in the operator's namespace. If it's not present, you may need to upgrade the Alluxio Operator. ```console $ kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=alluxio-collectinfo-controller NAME READY STATUS RESTARTS AGE alluxio-collectinfo-controller-cc49c56b6-wlw8k 1/1 Running 0 19s ``` #### Collecting the Snapshot By default, a `CollectInfo` resource is created with your cluster, performing a daily snapshot. You can also trigger a one-time collection or customize the schedule. To trigger a one-time collection, create a YAML file (`collect-now.yaml`): ```yaml apiVersion: k8s-operator.alluxio.com/v1 kind: CollectInfo metadata: name: one-time-snapshot namespace: alx-ns spec: scheduled: enabled: false # A single run is triggered when enabled is false ``` Then apply it: ```console $ kubectl apply -f collect-now.yaml ``` #### Accessing and Downloading the Snapshot The collected snapshots (as `.tar.gz` files) are stored in a volume mounted to the coordinator pod. ```console # 1. Get the coordinator pod name $ COORDINATOR_NAME=$(kubectl -n alx-ns get pod -l app.kubernetes.io/component=coordinator -o jsonpath="{.items[0].metadata.name}") # 2. List the snapshots inside the coordinator $ kubectl -n alx-ns exec -it ${COORDINATOR_NAME} -- ls /mnt/alluxio/metastore/collectinfo/ # 3. Copy the snapshots to your local machine $ kubectl -n alx-ns cp ${COORDINATOR_NAME}:/mnt/alluxio/metastore/collectinfo/ ./snapshots-output ``` ## 3. Common Issues and Recovery Procedures Here are step-by-step guides for recovering from common component failures. ### Coordinator Failure The coordinator handles metadata operations. It persists its state to the metastore and can recover upon restart. Kubernetes will automatically restart a failed coordinator pod. If job history is corrupted, unfinished jobs may be lost and need to be resubmitted. ### Worker Failure Alluxio is designed to be resilient to worker failures. If a worker pod fails, Kubernetes will restart it automatically. Data stored in cache on that worker will be lost, but this will not cause I/O operations to fail (though it may temporarily decrease performance as data is re-fetched). ### ETCD Failure Alluxio has a grace period (typically 24 hours) to tolerate an etcd failure without disrupting I/O. If the integrated etcd cluster fails and cannot be recovered by a simple pod restart, you may need to rebuild it. **Warning: This is a destructive operation and should only be performed as a last resort.** 1. **Shut down the Alluxio cluster:** `kubectl delete -f alluxio-cluster.yaml` 2. **Delete the original etcd PVCs:** `kubectl -n alx-ns delete pvc -l app.kubernetes.io/component=etcd` 3. **Clear etcd data on nodes:** Manually log into each Kubernetes node that hosted an etcd pod and delete the contents of the host path directory used by the etcd PV. 4. **Recreate the cluster:** `kubectl create -f alluxio-cluster.yaml`. The operator will provision a new, empty etcd cluster. 5. **Re-mount UFS paths:** If you were not using the `UnderFileSystem` CRD to manage mounts, you will need to manually re-add them using `alluxio fs mount`.