Troubleshooting

This guide provides a structured approach to troubleshooting your Alluxio cluster on Kubernetes. It covers everything from initial health checks to detailed diagnostics and recovery procedures for common issues.

1. Initial Health Checks

When you encounter an issue, start with these high-level checks to quickly assess the overall health of your Alluxio cluster and its dependencies.

Checking Component Status

Verify that all Alluxio and etcd pods are running and in a READY state. A Running status is not sufficient; the READY column should show that all containers in the pod are healthy.

Check the readiness of Alluxio coordinator pods:

kubectl -n alx-ns get pod -l app.kubernetes.io/component=coordinator

Check the readiness of Alluxio worker pods:

kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker

NAME                                      READY   STATUS    RESTARTS   AGE
alluxio-cluster-worker-59476bf8c5-lg4sc   1/1     Running   0          46h
alluxio-cluster-worker-59476bf8c5-vg6lc   1/1     Running   0          46h

Check the readiness of Alluxio FUSE pods (both DaemonSet and CSI):

kubectl -n alx-ns get pod -l 'app.kubernetes.io/component in (fuse, csi-fuse)'

NAME                                           READY   STATUS    RESTARTS   AGE
alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0   1/1     Running   0          57m

Check the readiness of the integrated etcd cluster:

kubectl -n alx-ns get pod -l 'app.kubernetes.io/component=etcd,app.kubernetes.io/instance=alluxio-cluster'

NAME                     READY   STATUS    RESTARTS   AGE
alluxio-cluster-etcd-0   1/1     Running   0          46h
alluxio-cluster-etcd-1   1/1     Running   0          46h
alluxio-cluster-etcd-2   1/1     Running   0          46h

You can also use this one-liner to get a readiness percentage for a specific component:

Example for workers:

kubectl -n alx-ns get pod -l app.kubernetes.io/component=worker -o jsonpath='{range .items[*]}{.status.containerStatuses[0].ready}{"\n"}{end}' | awk 'BEGIN{t=0}{s+=1;if($1=="true")t+=1}END{print t,"ready /",s,"expected =",t/s*100,"%"}'

2 ready / 2 expected = 100 %

Verifying UFS Connectivity

Ensure that Alluxio can communicate with the underlying storage system (UFS).

Run the ufsTest to check basic UFS operations:

./bin/alluxio exec ufsTest --path s3://your_bucket/test_path

Running test: createAtomicTest...
Passed the test! time: 5205ms
...
Tests completed with 0 failed.

Run the ufsIOTest to check UFS read/write throughput:

This example writes and reads a 512MB file with two threads:

./bin/alluxio exec ufsIOTest --path s3://test_bucket/test_path --io-size 512m --threads 2

{
  "readSpeedStat" : { ... },
  "writeSpeedStat" : { ... },
  "errors" : [ ],
  ...
}

A successful test with no errors indicates that the UFS is reachable and configured correctly.

Monitoring Key Metrics via Dashboard

The Grafana dashboard provides the quickest way to spot anomalies. Focus on these key areas:

• Liveliness: Look at the requests-per-second (RPS) for workers (irate(alluxio_data_access_bytes_count[5m])) and FUSE (alluxio_fuse_result). A sudden, unexpected spike or drop can indicate a problem.

• UFS Data Flow: Monitor the alluxio_ufs_data_access and alluxio_ufs_error metrics. An increase in errors is a clear sign of UFS connectivity or permission issues.

• Cache Hit Rate: A sudden drop in the overall cache hit rate can indicate that workers are unhealthy or that the data access pattern has changed unexpectedly.

2. Gathering Detailed Diagnostic Information

If initial health checks don't reveal the issue, you'll need to dig deeper by inspecting logs and collecting a full diagnostic snapshot.

Inspecting Logs

Alluxio Process Logs

Check the logs for specific error messages.

Get all logs from a specific pod (e.g., a worker):

kubectl -n alx-ns logs alluxio-cluster-worker-59476bf8c5-lg4sc

Filter for WARN or ERROR messages and show the line after the match:

kubectl -n alx-ns logs alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0 | grep -A 1 'WARN\|ERROR'

2024-07-04 17:29:53,499 ERROR HdfsUfsStatusIterator - Failed to list the path hdfs://localhost:9000/
java.net.ConnectException: Call From myhost/192.168.1.10 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

Check logs from a previously failed container:

kubectl -n alx-ns logs -p alluxio-cluster-worker-59476bf8c5-lg4sc

Kubernetes CSI Driver Logs

If you suspect issues with FUSE pod mounting, check the logs from the Alluxio CSI node plugin running on the same Kubernetes node as your application pod.

# 1. Get the node name where your application or FUSE pod is running
PODNS=alx-ns POD=alluxio-cluster-fuse-acee53e8f0a9-3gjbrdekk0
NODE_NAME=$(kubectl get pod -o jsonpath='{.spec.nodeName}' -n ${PODNS} ${POD})

# 2. Find the Alluxio CSI node plugin pod on that node
CSI_POD_NAME=$(kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=csi-nodeplugin --field-selector spec.nodeName=${NODE_NAME} -o jsonpath='{..metadata.name}')

# 3. Get the logs from the csi-nodeserver container
kubectl -n alluxio-operator logs -c csi-nodeserver ${CSI_POD_NAME}

Generating a Diagnostic Snapshot

For complex issues, the doctor tool gathers a comprehensive snapshot of your cluster's state, which is invaluable for offline analysis or for sharing with support.

The snapshot includes:

• Configuration files

• Hardware specifications of Kubernetes nodes

• Data from etcd (mounts, quotas, etc.)

• Logs from all Alluxio components

• Metrics over a specified time range

• Job service history

• Meta information

Prerequisites

Ensure the alluxio-doctor-controller is running in the operator's namespace. If it's not present, you may need to upgrade the Alluxio Operator.

kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=doctor-controller

NAME                                             READY   STATUS    RESTARTS   AGE
alluxio-doctor-controller-cc49c56b6-wlw8k        1/1     Running   0          19s

Collecting the Snapshot

By default, a Doctor is created with your cluster, performing a daily snapshot. You can also trigger a one-time collection or customize the schedule.

To trigger a one-time collection, create a YAML file (collect-now.yaml):

apiVersion: k8s-operator.alluxio.com/v1
kind: CollectInfo
metadata:
  name: one-time-snapshot
  namespace: alx-ns
spec:
  scheduled:
    enabled: false # A single run is triggered when enabled is false

Then apply it:

kubectl apply -f collect-now.yaml

Accessing and Downloading the Snapshot

The collected snapshots (as .tar.gz files) are stored in a volume mounted to the doctor-controller pod.

# 1. Get the doctor controller pod name
DOCTOR_NAME=$(kubectl -n alluxio-operator get pod -l app.kubernetes.io/component=doctor-controller -o jsonpath="{.items[0].metadata.name}")

# 2. List the snapshots inside the doctor controller
kubectl -n alluxio-operator exec -it ${DOCTOR_NAME} -- ls /data/doctor

# 3. Copy the snapshots to your local machine
kubectl -n alluxio-operator cp ${DOCTOR_NAME}:/data/doctor ./doctor

Snapshot Upload

By default, the diagnostic package collected by Doctor is stored only in the doctor-controller related storage within the cluster. Additionally, an optional feature is provided to automatically upload these diagnostic results to a dedicated S3 bucket maintained and analyzed by Alluxio.

You can enable this feature if you would like the Alluxio team to assist with analyzing your cluster's health.

How to enable:

1. Contact the Alluxio support team to request activation of this feature.

2. We will provide you with a dedicated awsKey and awsSecret.

3. Configure these credentials in the spec.upload field.

Once configured, the collected results will be securely uploaded. Our team will be able to access these reports to help you with data analysis, issue tracking, and proactively troubleshoot and prevent potential cluster problems.

Detailed Configuration

apiVersion: k8s-operator.alluxio.com/v1
kind: CollectInfo
metadata:
  name: example-doctor
  # Must be in the same namespace as the Alluxio cluster
  namespace: alx-ns
spec:
  # Configure scheduled collection
  scheduled:
    # false: run once immediately; true: enable scheduled execution
    enabled: false
    # Only effective when enabled: true
    cron: "0 0 * * *"
    # Only effective when enabled: true
    timeZone: "Asia/Shanghai"
    # Only effective when enabled: true, retention period for collected results
    expiration: "720h"

  # Information types to collect
  type:
    - all

  # Log collection configuration
  logs:
    # Collect logs from the past 1 hour (3600 seconds)
    # Note: If left empty, defaults to collecting logs from the past 1 day (86400 seconds)
    sinceSeconds: 3600
    # tail: 1000 # Alternatively, use tail to collect the last 1000 lines
    # sinceTime: "2025-11-12T06:00:00Z" # Alternatively, use sinceTime to collect logs after a specific time point

  # Metrics collection configuration
  metrics:
    # Collect metrics from the past 24 hours
    duration: 24h
    # Sampling interval of 5 minutes
    step: 5m

  # (Optional) Upload configuration
  # upload:
  #   account: test
  #   productionId: xxx # Optional
  #   awsKey: <alluxio-provided-key>
  #   awsSecret: <alluxio-provided-secret>

Field Details

spec.scheduled

Used to configure scheduled collection tasks.

• enabled (boolean):

  • false (default): Runs the collection once immediately after the CRD is applied (kubectl apply).

  • true: Enables scheduled collection, which will run cyclically according to the cron expression.

• cron (string): Only effective when enabled: true. Defines the Cron expression for the task execution schedule.

  • Example: "0 0 * * *" means run at midnight every day.

  • Syntax reference: Kubernetes Cron Syntax

• timeZone (string): Only effective when enabled: true. Defines the time zone for the cron expression.

  • Example: "Asia/Shanghai"

  • Time zone reference: Kubernetes Time Zones

• expiration (string): Only effective when enabled: true. Defines the retention period for the results of each scheduled collection, which will be automatically cleaned up after expiration.

  • Example: "720h" (retains for 30 days)

  • Format reference: Go Duration Format

spec.type

Defines the types of information to collect.

• all (default): If the type field is not specified, or is set to all, all the following information will be collected.

• Specify one or more types:

  • config: Alluxio configuration information.

  • hardware: Node hardware information.

  • etcd: Information stored in Etcd.

  • job-history: Job history records.

  • logs: Logs from components like Coordinator, Worker, FUSE, etc.

  • metrics: Prometheus metrics from Alluxio components.

spec.logs

Defines the scope of log collection, based on Kubernetes PodLogOptions. Reference: Kubernetes PodLogOptions

• Default behavior: If the logs field is left completely empty or not configured, the Operator will default to collecting logs from the past 1 day (86400 seconds).

• tail :

  • Collect the last N lines of each container's log.

  • Example: 1000 (collects the last 1000 lines).

• sinceSeconds :

  • Collect logs from N seconds ago to the present.

  • Example: 3600 (collects the past 1 hour).

• sinceTime :

  • Collect logs after a specific absolute time point (RFC3339 format).

  • Example: "2025-11-12T06:00:00Z" (collects logs after 6:00 AM UTC on November 12th).

• Collection Rules:

  1. tail has the highest priority. When tail is set, the sinceSeconds and sinceTime fields are ignored.

  2. Only when tail is not set, sinceSeconds or sinceTime will take effect, but only one of them can be set.

  3. Default behavior (if all are left empty), it will collect logs from the past 1 day (86400 seconds).

spec.metrics

Defines the scope for Prometheus metrics collection.

• duration (string): The duration to look back from "now" for collection.

  • Example: 24h (collects metrics from the past 24 hours).

• step (string): The time interval for collection (sampling precision).

  • Example: 1m (samples once per minute).

spec.upload (Optional)

Provides a way to automatically upload the collected results (compressed package) to an AWS S3 bucket provided by Alluxio.

• Default behavior: If the upload block is not configured, the collected information ("doctor" results) will not be automatically uploaded. You will need to manually retrieve the results from the Operator's Pod or its mounted storage. (Note: You need to confirm and specify the local storage location of the results based on your actual situation).

• account (string): Required. Account information provided by Alluxio.

• productionId (string): Optional. A product ID used to identify the customer environment.

• awsKey (string): Required. AWS Access Key provided by Alluxio.

• awsSecret (string): Required. AWS Secret Key provided by Alluxio.

3. Common Issues and Recovery Procedures

Here are step-by-step guides for recovering from common component failures.

Coordinator Failure

The coordinator runs the Alluxio job service, which is responsible for managing asynchronous jobs like distributed loads for preloading cache. It persists its job history and can recover its state upon restart. Kubernetes will automatically restart a failed coordinator pod. If the job history is corrupted, any unfinished jobs may be lost and will need to be resubmitted.

Worker Failure

Alluxio is designed to be resilient to worker failures. If a worker pod fails, Kubernetes will restart it automatically. Data stored in cache on that worker will be lost, but this will not cause I/O operations to fail (though it may temporarily decrease performance as data is re-fetched).

FUSE Failure

If a FUSE pod crashes or becomes unresponsive, it will be automatically restarted by its controller (either a DaemonSet or the CSI driver). If a FUSE pod is hung, you can force a restart:

# Manually delete the pod to trigger a restart
kubectl delete pod <fuse-pod-name>

ETCD Failure

Alluxio has a grace period (typically 24 hours) to tolerate an etcd failure without disrupting I/O. If the integrated etcd cluster fails and cannot be recovered by a simple pod restart, you may need to rebuild it.

Warning: This is a destructive operation and should only be performed as a last resort.

1. Shut down the Alluxio cluster: kubectl delete -f alluxio-cluster.yaml

2. Delete the original etcd PVCs: kubectl -n alx-ns delete pvc -l app.kubernetes.io/component=etcd

3. Clear etcd data on nodes: Manually log into each Kubernetes node that hosted an etcd pod and delete the contents of the host path directory used by the etcd PV.

4. Recreate the cluster: kubectl create -f alluxio-cluster.yaml. The operator will provision a new, empty etcd cluster.

5. Re-mount UFS paths: If you were not using the UnderFileSystem CRD to manage mounts, you will need to manually re-add them using alluxio fs mount.