> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/administration/monitoring-alluxio.md). # Monitoring Alluxio exposes metrics in the [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/), enabling integration with standard monitoring stacks. This guide covers Prometheus setup, Grafana dashboard import, alert rules, and direct metric queries for both Kubernetes (Operator) and Docker/Bare-Metal deployments. ## Prometheus Setup {% tabs %} {% tab title="Kubernetes (Operator)" %} The Alluxio Operator deploys a Prometheus instance alongside your cluster automatically. No manual configuration is required. Verify Prometheus is running: ```shell kubectl -n alx-ns get pod -l app.kubernetes.io/component=prometheus ``` ```console NAME READY STATUS RESTARTS AGE alluxio-cluster-prometheus-6f697b6db8-sbvvg 1/1 Running 0 2m ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} Run Prometheus on the coordinator node with a static scrape configuration pointing at all Alluxio components. **Step 1: Create the Prometheus config** ```shell mkdir -p ~/monitoring/prometheus ``` Create `~/monitoring/prometheus/prometheus.yml`: ```yaml global: scrape_interval: 60s scrape_configs: - job_name: "coordinator" static_configs: - targets: [":19999"] - job_name: "workers" static_configs: - targets: [":30000", ":30000"] - job_name: "fuse" static_configs: - targets: [":49999"] ``` Add one entry per worker under `targets`. **Step 2: Start Prometheus** ```shell docker run -d --net=host --name=prometheus \ -v ~/monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus --config.file=/etc/prometheus/prometheus.yml ``` **Step 3: Verify targets are UP** {% hint style="info" %} Prometheus scrapes on the `scrape_interval` configured above (60 s). Targets show `unknown` until the first scrape completes — wait up to 60 seconds before checking. {% endhint %} Open `http://localhost:9090/targets` in a browser (or via SSH tunnel — see [Grafana Setup](#grafana-setup) for access options), or query the API: ```shell curl -s 'http://localhost:9090/api/v1/targets' | \ python3 -c "import sys,json; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]" ``` ```console coordinator up workers up fuse up ``` {% endtab %} {% endtabs %} ### Kubernetes: Bring Your Own Prometheus If your cluster already has a Prometheus instance, you can disable the Operator-managed one and use Kubernetes service discovery instead. Disable the Operator-managed Prometheus: ```yaml apiVersion: k8s-operator.alluxio.com/v1 kind: AlluxioCluster spec: prometheus: enabled: false ``` Add the following scrape config to your existing `prometheus.yml` to automatically discover Alluxio pods by annotation: ```yaml scrape_configs: - job_name: 'alluxio-components' kubernetes_sd_configs: - role: pod relabel_configs: # Keep only pods with prometheus.io/scrape=true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true # Keep only Alluxio components - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] action: keep regex: alluxio # Use the annotated metrics path, default to /metrics - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) # Use the annotated port - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ # Set job label from the component name - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component] action: replace target_label: job # Propagate cluster name for multi-cluster Grafana dashboards - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] action: replace target_label: cluster_name - job_name: 'etcd' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] action: keep regex: etcd - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - target_label: job replacement: etcd - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] action: replace target_label: cluster_name ``` Your Alluxio pods must carry the following labels and annotations for discovery to work: ```yaml # Example metadata for an Alluxio worker pod metadata: labels: app.kubernetes.io/name: alluxio app.kubernetes.io/component: worker # or coordinator, fuse annotations: prometheus.io/scrape: "true" prometheus.io/port: "30000" # 19999 for coordinator, 49999 for fuse prometheus.io/path: "/metrics/" ``` ## Grafana Setup {% tabs %} {% tab title="Kubernetes (Operator)" %} The Operator deploys Grafana automatically alongside your cluster. #### Access via Port Forwarding (Recommended) ```shell kubectl -n alx-ns port-forward \ $(kubectl -n alx-ns get pod -l app.kubernetes.io/component=grafana -o jsonpath="{.items[0].metadata.name}") \ 3000:3000 ``` Then open `http://localhost:3000` in your browser. #### Access via Node Hostname If Kubernetes nodes are directly accessible on your network, look up the node where Grafana is scheduled: ```shell kubectl -n alx-ns get pod \ $(kubectl -n alx-ns get pod -l app.kubernetes.io/component=grafana --no-headers -o custom-columns=:metadata.name) \ -o jsonpath='{.spec.nodeName}' ``` Then access Grafana at `http://:8080/`. #### Disabling the Default Grafana To use your own Grafana instance, disable the Operator-managed one: ```yaml apiVersion: k8s-operator.alluxio.com/v1 kind: AlluxioCluster spec: grafana: enabled: false ``` {% hint style="info" %} Prometheus is a core component of the Operator deployment and cannot be disabled independently. {% endhint %} {% endtab %} {% tab title="Docker / Bare-Metal" %} Run Grafana on the coordinator node alongside Prometheus. Pre-provision the Prometheus datasource so no manual setup is needed after Grafana starts. **Step 1: Create the datasource provisioning file** ```shell mkdir -p ~/monitoring/grafana/provisioning/datasources ``` Create `~/monitoring/grafana/provisioning/datasources/prometheus.yml`: ```yaml apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://localhost:9090 isDefault: true access: proxy editable: true ``` **Step 2: Start Grafana** ```shell docker run -d --net=host --name=grafana \ -v ~/monitoring/grafana/provisioning:/etc/grafana/provisioning \ -e GF_SECURITY_ADMIN_USER=admin \ -e GF_SECURITY_ADMIN_PASSWORD=grafana \ grafana/grafana ``` **Step 3: Access Grafana** {% hint style="info" %} Ports 3000 (Grafana) and 9090 (Prometheus) are closed by default in EC2 Security Groups. Either open them in your Security Group, or use an SSH tunnel: ```shell ssh -L 3000:localhost:3000 -L 9090:localhost:9090 user@ ``` Then access Grafana at `http://localhost:3000`. {% endhint %} If ports are open in your Security Group, access Grafana directly at `http://:3000` (login: `admin` / `grafana`). {% endtab %} {% endtabs %} ## Dashboard Import Download the official Alluxio dashboard template and import it into Grafana: ```shell wget -O /tmp/alluxio-dashboard.json \ https://alluxio-binaries.s3.amazonaws.com/artifactsBundle/ee/AI-3.8-15.1.0/alluxio-ai-dashboard-template.json ``` In Grafana: **Dashboards → Import → Upload JSON file** → select `/tmp/alluxio-dashboard.json` → select **Prometheus** as the data source → click **Import**. For detailed import options, see the [Grafana import guide](https://grafana.com/docs/grafana/latest/dashboards/export-import/#importing-a-dashboard). ### Understanding the Dashboard
* The **Cluster** section gives a high-level summary of the cluster status. * The **Process** section shows resource consumption (CPU, memory) and JVM metrics for each component. * Additional sections provide detailed metrics for the coordinator, workers, and cache. ## Alert Rules The queries below can be used to build Prometheus alert rules or Grafana alert panels. Thresholds are recommended starting points — tune them to your workload and cluster size. ### Process Availability — ETCD | Field | Value | | ------------------ | ------------------------------------------------ | | Component | Process Availability - ETCD | | Metric | `etcd_server_has_leader` | | Metric Explanation | Shows if each etcd member currently has a leader | | Query | `sum(etcd_server_has_leader{job="etcd"})` | | Query Explanation | Sums all members that currently have a leader | | Trigger Condition | value < 3 | | Threshold/Value | 3 members expected | | Meaning | One or more etcd pods are down or quorum is lost | | Note | | | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------- | | Component | Process Availability - ETCD | | Metric | `etcd_server_leader_changes_seen_total` | | Metric Explanation | Counts how many times leader has changed | | Query | `changes(etcd_server_leader_changes_seen_total{job="etcd"}[5m])` | | Query Explanation | Calculates the number of leader changes (elections) that occurred within the last 5 minutes | | Trigger Condition | > 0 for 5+ min | | Threshold/Value | Any change > 0 | | Meaning | Leader flapping; indicates etcd instability or network issues | | Note | Query needs to be modified on the dashboard from 1d to 5m | ### Process Availability — Worker Count | Field | Value | | ------------------ | --------------------------------------------------------- | | Component | Process Availability - Worker count | | Metric | `up{job="worker"}` | | Metric Explanation | Shows how many workers are alive (responding to scrapes) | | Query | `sum(up{job="worker"})` | | Query Explanation | Counts the number of live worker targets | | Trigger Condition | value < desired worker count | | Threshold/Value | < desired worker count | | Meaning | One or more workers are down or not responding | | Note | Set desired worker count to match production cluster size | ### Process Resource | Field | Value | | ------------------ | -------------------------------------------------------------------------------------------------------------------- | | Component | Process Resource | | Metric | `jvm_memory_used_bytes` | | Metric Explanation | Shows current JVM heap usage as % of max | | Query | `jvm_memory_used_bytes{area="heap"}/jvm_memory_max_bytes{area="heap"}` | | Query Explanation | Calculates current heap usage as a percentage of the maximum heap | | Trigger Condition | > 0.75 for 5+ min | | Threshold/Value | 75–80% | | Meaning | Component is using a high percentage of its heap memory, indicating potential memory pressure or impending GC thrash | | Note | Applies to all components (coordinator, workers, fuse, etc.) | | Field | Value | | ------------------ | ----------------------------------------------------------------- | | Component | Process Resource | | Metric | `jvm_gc_collection_seconds_sum` | | Metric Explanation | Time spent in old GC collections | | Query | `rate(jvm_gc_collection_seconds_sum{gc="G1 Old Generation"}[5m])` | | Query Explanation | Calculates time spent in old/full GC over 5 minutes | | Trigger Condition | > 5s/min for 5+ min | | Threshold/Value | > 0.083 | | Meaning | JVM doing frequent full GCs → major pause risk | | Note | Combine with old GC count to confirm | | Field | Value | | ------------------ | ------------------------------------------------------------------- | | Component | Process Resource | | Metric | `jvm_gc_collection_seconds_count` | | Metric Explanation | Frequency of old GC collections | | Query | `rate(jvm_gc_collection_seconds_count{gc="G1 Old Generation"}[5m])` | | Query Explanation | Calculates number of old/full GCs per minute | | Trigger Condition | > 1/min for 5+ min | | Threshold/Value | > 1 | | Meaning | JVM doing many full GCs, likely due to memory pressure | | Note | Early memory pressure warning | | Field | Value | | ------------------ | ------------------------------------------------------------------- | | Component | Process Resource | | Metric | `jvm_gc_collection_seconds_sum` | | Metric Explanation | Time spent in young GC collections | | Query | `rate(jvm_gc_collection_seconds_sum{gc="G1 Young Generation"}[5m])` | | Query Explanation | Calculates time spent in young GC over 5 minutes | | Trigger Condition | > 10s/min for 5+ min | | Threshold/Value | > 0.166 | | Meaning | High GC overhead slowing throughput | | Note | Only alert if persistent | | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------ | | Component | Process Resource | | Metric | `process_cpu_seconds_total` | | Metric Explanation | Measures total user + system CPU time consumed by the process | | Query | `irate(process_cpu_seconds_total{job=~"$service",instance=~"$instance",cluster_name=~"$cluster"}[5m])` | | Query Explanation | Calculates the per-second CPU usage rate over 5 minutes | | Trigger Condition | stays consistently high for 5+ min | | Threshold/Value | > 80% of 1 CPU core (≈ 0.8) | | Meaning | Process is CPU bound or stuck consuming full CPU | | Note | Tune threshold based on node vCPU cores; alert if usage is flat and near saturation | ### Cache — Cache Hit Rate | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Component | Cache - Cache Hit Rate | | Metric | `alluxio_cached_data_read_bytes_total` & `alluxio_missed_data_read_bytes_total` | | Metric Explanation | Measures how much read data was served from cache vs fetched from UFS | | Query | `sum(irate(alluxio_cached_data_read_bytes_total{job="worker",cluster_name=~"$cluster"}[5m])) / (sum(irate(alluxio_cached_data_read_bytes_total{job="worker",cluster_name=~"$cluster"}[5m])) + sum(irate(alluxio_missed_data_read_bytes_total{job="worker",cluster_name=~"$cluster"}[5m])))` | | Query Explanation | Calculates cache hit ratio over 5 minutes | | Trigger Condition | cache hit % stays low for 5+ min | | Threshold/Value | < 80% | | Meaning | High UFS reads, cache not being utilized effectively | | Note | Adjust threshold based on workload (e.g. 70–90%) | ### Cache — Utilization | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Component | Cache - Utilization | | Metric | `alluxio_cached_storage_bytes` & `alluxio_cached_capacity_bytes` | | Metric Explanation | Shows how much of the configured cache capacity is currently used | | Query | `sum(alluxio_cached_storage_bytes{job="worker",cluster_name=~"$cluster"}) / sum(alluxio_cached_capacity_bytes{job="worker",cluster_name=~"$cluster"})` | | Query Explanation | Calculates current used/total cache ratio | | Trigger Condition | > 0.85 (warning), > 0.95 (critical) for 5+ min | | Threshold/Value | 85–95% utilization | | Meaning | Cache is nearly full, risk of eviction thrash or write failures | | Note | Adjust thresholds based on cluster size and workload pattern | ### Cache — Eviction Correlation | Field | Value | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Component | Cache - Cache Eviction - Correlation | | Metric | `alluxio_cached_evicted_data_bytes_total` + `alluxio_block_store_used_bytes` | | Metric Explanation | Tracks evicted bytes and current cache usage to detect cache pressure | | Query | `(sum(irate(alluxio_cached_evicted_data_bytes_total{job="worker"}[5m])) > 0) and ((sum(alluxio_block_store_used_bytes{job="worker"}) / sum(alluxio_block_store_capacity_bytes{job="worker"})) > 0.8)` | | Query Explanation | Checks if evictions are occurring while cache usage is above 80% | | Trigger Condition | Evictions > 0 while usage > 80% for 5+ minutes | | Threshold/Value | Usage > 80% and Evictions > 0 | | Meaning | Indicates cache thrashing or pressure (evictions happening despite high cache utilization) | | Note | Needs to be created manually as a new panel | ### FUSE — UFS Fallback | Field | Value | | ------------------ | --------------------------------------------------------------------------------------------------- | | Component | Fuse - UFS Fallback | | Metric | `alluxio_ufs_data_access_bytes_total` | | Metric Explanation | Tracks read traffic from Fuse pods going directly to the UFS (bypassing Alluxio cache) | | Query | `irate(alluxio_ufs_data_access_bytes_total{job="fuse",method="read",cluster_name=~"$cluster"}[5m])` | | Query Explanation | Calculates Fuse-driven UFS read throughput over 5 minutes | | Trigger Condition | sustained Fuse UFS read traffic increases | | Threshold/Value | >10 MiB/s sustained for >5m | | Meaning | Fuse clients are bypassing Alluxio cache, high fallback | | Note | Correlate with cache hit % and request rate; fallback >10–20 MiB/s usually worth investigating | ### Read Throughput | Field | Value | | ------------------ | ---------------------------------------------------------------------------------------------------------- | | Component | Read Throughput | | Metric | `alluxio_data_throughput_bytes_total` | | Metric Explanation | Measures read throughput served by workers | | Query | `sum(irate(alluxio_data_throughput_bytes_total{job="worker",method="read",cluster_name=~"$cluster"}[5m]))` | | Query Explanation | Calculates worker read throughput over 5m | | Trigger Condition | worker read throughput drops | | Threshold/Value | < set baseline (e.g. < 10 MiB/s) while UFS read goes up | | Meaning | Cache not serving data, workload hitting UFS | | Note | Tune threshold based on normal workload pattern | ### Data — Read Request Rate | Field | Value | | ------------------ | ------------------------------------------------------------------------ | | Component | Data | | Metric | `alluxio_data_access_bytes_count{method="read"}` | | Metric Explanation | Counts the number of read operations (requests) served by workers | | Query | `irate(alluxio_data_access_bytes_count{method="read",job="worker"}[5m])` | | Query Explanation | Calculates read request rate (req/s) over 5 minutes | | Trigger Condition | Alert when rate drops to 0 while workload is expected | | Threshold/Value | near 0 for > 5 min | | Meaning | Worker not serving data — possible worker crash or cache unavailable | | Note | Correlate with workload schedule to avoid false positives | ### License — Expiration | Field | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | Component | License - Expiration | | Metric | `alluxio_license_expiration_date` | | Metric Explanation | Shows the UNIX timestamp when the Alluxio license will expire | | Query | `(max by (cluster_name) (alluxio_license_expiration_date) - time()) / 86400` | | Query Explanation | Calculates the number of days remaining until license expiration by subtracting current time from the license expiry timestamp | | Trigger Condition | < 30 (Warning), < 7 (Critical) | | Threshold/Value | 30 days, 7 days | | Meaning | License is about to expire; renew before it lapses | | Note | Needs to be created manually as a new panel | ### License — Version Mismatch | Field | Value | | ------------------ | --------------------------------------------------------------------------- | | Component | License - Version Mismatch | | Metric | `alluxio_version_info` | | Metric Explanation | Shows the version of each running Alluxio component (via version label) | | Query | `count(count by (version) (alluxio_version_info)) > 1` | | Query Explanation | Checks if more than one unique Alluxio version is running across components | | Trigger Condition | > 1 | | Threshold/Value | More than 1 version | | Meaning | Version mismatch between Alluxio components | | Note | Needs to be created manually as a new panel | ## Querying Metrics Directly For advanced analysis or debugging, query Prometheus or component endpoints directly. {% tabs %} {% tab title="Kubernetes (Operator)" %} Open a shell into the Prometheus pod: ```shell kubectl -n alx-ns exec -it \ $(kubectl -n alx-ns get pod -l app.kubernetes.io/component=prometheus --no-headers -o custom-columns=:metadata.name) \ -- /bin/sh ``` Then use `promtool` to run instant queries: ```shell # List all available Alluxio metrics promtool query instant http://localhost:9090 'count({__name__=~".+"}) by (__name__)' | grep alluxio_ # Check total cache capacity across all workers promtool query instant http://localhost:9090 'alluxio_cached_capacity_bytes' # Example output: # alluxio_cached_capacity_bytes{instance="worker:30000", job="worker"} => 10737418240 @[...] ``` Query component endpoints directly from within a pod: ```shell # Coordinator metrics kubectl -n alx-ns exec alluxio-cluster-coordinator-0 -- curl -s http://localhost:19999/metrics/ | head -20 # Worker metrics kubectl -n alx-ns exec alluxio-cluster-worker-0 -- curl -s http://localhost:30000/metrics/ | head -20 ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} Query via the Prometheus HTTP API from the coordinator host: ```shell # Check cache capacity for all workers curl -s 'http://localhost:9090/api/v1/query?query=alluxio_cached_capacity_bytes' | python3 -m json.tool # Check live worker count curl -s 'http://localhost:9090/api/v1/query?query=sum(up{job="workers"})' | python3 -m json.tool ``` Query component endpoints directly: ```shell # Coordinator metrics (from coordinator host) curl http://localhost:19999/metrics/ # Worker metrics (from worker host) curl http://localhost:30000/metrics/ # FUSE metrics (from FUSE host) curl http://localhost:49999/metrics/ ``` {% endtab %} {% endtabs %} Refer to the [Metrics Reference](/ee-ai-en/reference/metrics.md) for a complete list of available metrics and their descriptions. ## Datadog Integration Datadog can ingest metrics directly from Alluxio's Prometheus endpoints. 1. Ensure your Datadog agent can reach the Alluxio metrics ports: `19999` (coordinator), `30000` (workers), `49999` (FUSE). 2. Add the following to your `conf.d/prometheus.d/conf.yaml`: ```yaml instances: - prometheus_url: http://:19999/metrics namespace: alluxio metrics: - "*" - prometheus_url: http://:30000/metrics namespace: alluxio metrics: - "*" # Add one entry per worker ``` This configuration instructs the Datadog agent to scrape and forward all Alluxio metrics.