> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/administration/managing-worker.md). # Worker Configuration This guide covers configuration for individual Alluxio workers: storage backends, capacity sizing, resource limits, JVM tuning, and network binding. For hash-ring-related worker operations (adding, removing, restarting, identity persistence), see [Hash Ring and Worker Lifecycle](/ee-ai-en/administration/managing-ring.md). For cluster-wide operations (scaling, upgrades, coordinator, UFS, multi-tenancy), see [Cluster Management](/ee-ai-en/administration/managing-alluxio.md). ## 1. Worker Storage Each worker caches data in a local [page store](https://documentation.alluxio.io/ee-ai-en/administration/pages/iRTxT4smG58AwFmCihOx#id-5.-worker-storage-the-page-store). This section covers the storage backend choice, capacity sizing, and disk layout. ### Configuring Page Store Location {% tabs %} {% tab title="Kubernetes (Operator)" %} The Operator supports two page store backends. **Default (hostPath):** The worker writes cache to the node's filesystem at `/mnt/alluxio/pagestore`. ```yaml spec: worker: pagestore: # Defaults to hostPath: /mnt/alluxio/pagestore on the node's filesystem. size: 100Gi reservedSize: 10Gi ``` **PVC-backed:** To persist worker cache data across pod restarts or rescheduling, specify a PersistentVolumeClaim (PVC) for the page store. ```yaml spec: worker: pagestore: type: persistentVolumeClaim storageClass: "" # defaults to "standard"; empty string = static binding size: 100Gi reservedSize: 10Gi ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} Set the page store directory and size via Alluxio properties. On Docker, pass them through `ALLUXIO_JAVA_OPTS` and mount the host directory as a volume so the cache survives container recreation: ```shell docker run ... \ -v /data/alluxio-cache:/data/alluxio-cache \ -e ALLUXIO_JAVA_OPTS=" \ -Dalluxio.worker.page.store.dirs=/data/alluxio-cache \ -Dalluxio.worker.page.store.sizes=" \ alluxio/alluxio-enterprise worker ``` On bare-metal, set the same properties in `conf/alluxio-site.properties`: ```properties alluxio.worker.page.store.dirs=/data/alluxio-cache alluxio.worker.page.store.sizes= ``` {% endtab %} {% endtabs %} ### Sizing the Page Store * `size`: Per-worker cache capacity. Must not exceed available disk on the worker node. * `reservedSize`: Space reserved for internal operations (temporary page writes, metadata caching). Set to \~10% of `size`, typically 10–100 GiB. * Ensure `size + reservedSize ≤ available disk space`. Cloud providers advertise disk size in GB (base-10), while Kubernetes `Gi` is base-2. A "1000 GB" EBS volume provides \~931 GiB. Set `size` to \~90% of actual available space (check with `df -h `) to leave room for filesystem overhead and `reservedSize`. A too-large `size` causes workers to crash with `quota (NNN) exceeds the total disk space`. ### Multi-Disk Configuration For nodes with multiple data disks, configure the page store to span all of them — this distributes page I/O across disks and increases aggregate throughput. Use comma-separated paths and sizes: ```yaml spec: worker: pagestore: hostPath: /mnt/disk1/alluxio/pagestore,/mnt/disk2/alluxio/pagestore size: 800Gi,800Gi reservedSize: 100Gi ``` Each directory must map to a separate physical disk — colocating multiple page store directories on the same disk provides no benefit. An alternative is to use RAID 0 at the OS level to present multiple disks as a single logical volume, then configure a single `hostPath`. This simplifies the Alluxio configuration but couples the lifetime of all disks — a single disk failure loses the entire array. ### Heterogeneous Workers The `workerGroups` mechanism described in this section is specific to the Kubernetes Operator. On Docker/bare-metal deployments, heterogeneous workers are achieved by running each worker with its own `alluxio-site.properties` file that specifies the per-node page store paths and sizes. When a cluster has workers with different disk specifications (e.g., one group with a 1 TB disk, another with two 800 GB disks), use `workerGroups` to define distinct configurations per group. **Step 1: Group and label nodes:** ```shell # Label nodes with one disk kubectl label nodes apps.alluxio.com/disks=1 # Label nodes with two disks kubectl label nodes apps.alluxio.com/disks=2 ``` **Step 2: Define worker groups and enable capacity-aware hash ring.** For heterogeneous clusters, set `alluxio.user.worker.selection.policy.consistent.hash.provider.impl` to `CAPACITY` so workers with more storage receive a proportionally larger share of data. For details on this property, see [Optimizing for Heterogeneous Workers](/ee-ai-en/administration/managing-ring.md#optimizing-for-heterogeneous-workers). ```yaml apiVersion: k8s-operator.alluxio.com/v1 kind: AlluxioCluster spec: properties: alluxio.user.worker.selection.policy.consistent.hash.provider.impl: CAPACITY # Common configurations for all workers worker: resources: limits: memory: 40Gi requests: memory: 36Gi jvmOptions: ["-Xmx20g", "-Xms20g", "-XX:MaxDirectMemorySize=16g"] # Define specific configurations for each worker group workerGroups: - worker: count: 10 nodeSelector: apps.alluxio.com/disks: 1 pagestore: hostPath: /mnt/disk1/alluxio/pagestore size: 1Ti reservedSize: 100Gi - worker: count: 12 nodeSelector: apps.alluxio.com/disks: 2 pagestore: hostPath: /mnt/disk1/alluxio/pagestore,/mnt/disk2/alluxio/pagestore size: 800Gi,800Gi reservedSize: 100Gi ``` {% hint style="info" %} While this provides flexibility, it is crucial to ensure consistency within each worker group. Misconfigurations can lead to unexpected errors. {% endhint %} ## 2. Resource and JVM Tuning {% tabs %} {% tab title="Kubernetes (Operator)" %} Configure per-component resource limits and JVM options in `alluxio-cluster.yaml`: ```yaml apiVersion: k8s-operator.alluxio.com/v1 kind: AlluxioCluster spec: worker: count: 2 resources: # For production workers, set requests equal to limits so the pod # runs in the Guaranteed QoS class and is the last to be evicted # under node pressure. limits: cpu: "12" memory: "36Gi" requests: cpu: "12" memory: "36Gi" jvmOptions: - "-Xmx22g" - "-Xms22g" - "-XX:MaxDirectMemorySize=10g" coordinator: resources: # Coordinator should also run as Guaranteed QoS — set requests == limits. limits: cpu: "12" memory: "36Gi" requests: cpu: "12" memory: "36Gi" jvmOptions: - "-Xmx4g" - "-Xms1g" ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} On Docker, set JVM options through `ALLUXIO_JAVA_OPTS` and the container resource limits via `--memory` and `--cpus`: ```shell docker run ... \ --memory=36g \ --cpus=12 \ -e ALLUXIO_JAVA_OPTS=" \ -Xmx22g -Xms22g -XX:MaxDirectMemorySize=10g" \ alluxio/alluxio-enterprise worker ``` On bare-metal, set JVM options in `conf/alluxio-env.sh`: ```shell ALLUXIO_WORKER_JAVA_OPTS+=" -Xmx22g -Xms22g -XX:MaxDirectMemorySize=10g" ALLUXIO_COORDINATOR_JAVA_OPTS+=" -Xmx4g -Xms1g" ``` {% endtab %} {% endtabs %} ### Memory Limit Formula ``` memory limit ≥ -Xmx + -XX:MaxDirectMemorySize + 2–4 GiB (JVM overhead) ``` For the worker config above (`-Xmx22g`, `-XX:MaxDirectMemorySize=10g`): minimum limit is 22 + 10 + 2 = 34 GiB, set to 36 GiB in the example. {% hint style="info" %} If `-XX:MaxDirectMemorySize` is omitted, the JVM defaults it to the same value as `-Xmx`, so the container limit typically needs to be 2.5× `-Xmx` or more. {% endhint %} ### Diagnosing OOM If a worker is killed due to OOM (exit code 137), confirm the cause using the commands below. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # Check pod events kubectl describe pod -n alx-ns # Look for "OOMKilled" or "Exit Code: 137" # Check the previous container's logs (before crash) kubectl logs -n alx-ns --previous | tail -50 # Look for "OutOfMemoryError" or "java.lang.OutOfMemoryError" ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # Docker: inspect the exited container docker inspect --format='{{.State.OOMKilled}} {{.State.ExitCode}}' # "true 137" indicates OOM # Tail worker logs for OutOfMemoryError tail -100 /opt/alluxio/logs/worker.log | grep -i OutOfMemoryError ``` {% endtab %} {% endtabs %} | Symptom | Root Cause | Fix | | -------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------ | | `Exit Code 137`, no Java error | Container limit exceeded — killed by Linux OOM killer | Increase `resources.limits.memory` | | `java.lang.OutOfMemoryError: Java heap space` | `-Xmx` too small | Increase `-Xmx` and raise container limit accordingly | | `java.lang.OutOfMemoryError: Direct buffer memory` | `-XX:MaxDirectMemorySize` too small | Increase `-XX:MaxDirectMemorySize` and raise container limit accordingly | ### Diagnosing Stalls If peers log redirect or connect timeouts to a worker and reads fall back to the UFS while the worker pod stays `READY`, the worker JVM is probably stopping for seconds at a time, either for garbage collection (GC) or at a safepoint, where the JVM stops all Java threads for some internal operations. Check for that before tuning the network: rising TCP listen-queue drops on the node are a result of the pauses, not their cause. **1. Confirm the pauses.** The worker logs every pause longer than 1 second as `JVM paused ms` (at WARN above 10 seconds). The line after it says whether a GC ran during the pause: ```shell # Kubernetes kubectl logs -n alx-ns | grep -A1 "JVM paused" # Docker / bare-metal grep -A1 "JVM paused" /opt/alluxio/logs/worker.log ``` `GC list:` means a GC ran during the pause; `No GCs detected` means the JVM stopped for something else. GC metrics alone cannot tell these apart, because they do not count non-GC pauses. **2. Collect a GC and safepoint log**, over a few hours: a short capture can single out the wrong cause. Add the option to the worker JVM options, which takes effect on the next restart: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```yaml spec: worker: jvmOptions: # jvmOptions replaces the default list: keep your heap, direct-memory, and other flags - "-Xlog:gc*,safepoint:file=/opt/alluxio/logs/gc.log:time,uptime:filecount=5,filesize=50M" ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # conf/alluxio-env.sh; on Docker, append the flag to ALLUXIO_JAVA_OPTS instead ALLUXIO_WORKER_JAVA_OPTS+=" -Xlog:gc*,safepoint:file=/opt/alluxio/logs/gc.log:time,uptime:filecount=5,filesize=50M" ``` {% endtab %} {% endtabs %} Or turn the same log on in a running worker, without a restart. The setting is lost when the worker restarts. ```shell # Kubernetes: open a shell in the worker container kubectl exec -it -n alx-ns -- bash # Find the worker JVM's process ID: the line ending in alluxio.worker.AlluxioWorker jcmd -l jcmd VM.log output="file=/opt/alluxio/logs/gc.log" output_options="filecount=5,filesize=50M" what="gc*,safepoint" decorators="time,uptime" ``` On Kubernetes, copy the log out after a few hours with `kubectl cp alx-ns/:/opt/alluxio/logs/gc.log gc.log`. **3. Match the log to a cause.** Each GC appears as a group of lines with the same `GC()` number: the pause and its total time, the time of each phase, and the CPU time its threads used. Each safepoint appears as one `Safepoint` line, with its times in nanoseconds. A stall looks like this: ``` GC(189) Pause Young (Normal) (G1 Evacuation Pause) 21G->13G(36G) 7040.519ms GC(189) Pre Evacuate Collection Set: 6998.1ms GC(189) Evacuate Collection Set: 36.2ms GC(189) User=0.67s Sys=0.01s Real=7.04s Safepoint "Cleanup", Time since last: 1000120313 ns, Reaching safepoint: 4812339871 ns, At safepoint: 25163042 ns, Total: 4837502913 ns ``` Here a 7-second young GC spent almost all of its time in `Pre Evacuate Collection Set`, and a `Cleanup` safepoint took 4.8 seconds to reach. Start from the rows that match what step 1 showed. | Step 1 shows | Symptom in the GC and safepoint log | Root Cause | Fix | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `GC list` | `Pause Full` or `To-space exhausted`, or long `Pause Young` with most of the time in `Evacuate Collection Set` | Heap too small for the live data | Increase `-Xmx` and raise container limit accordingly | | `GC list` | Long GC pause whose `User` CPU time is far below `Real` | GC threads held off the CPU by the container's CPU limit (`nr_throttled` rising in the cgroup's `cpu.stat`) | Raise the CPU limit | | `GC list` or `No GCs detected` | Long `Pause Young` with most of the time in `Pre Evacuate Collection Set`, or `Safepoint` lines whose `Reaching safepoint` takes seconds | Threads writing the JVM's mmapped `/tmp/hsperfdata_/` file block on the container's dirty-page limit | Add `-XX:+PerfDisableSharedMem` to the worker JVM options | | `No GCs detected` | Seconds-long `Safepoint "FindDeadlocks"` lines at every metrics scrape | `alluxio.metrics.jvm.deadlock.detection.enabled=true` | Leave the property at its default, `false` | For the hsperfdata case, the dirty-page limit is the node's `vm.dirty_bytes` or `vm.dirty_ratio`, scaled down to each container's memory cgroup; an absolute `vm.dirty_bytes` can leave a container only a few MB. To confirm, check that `file_dirty` in the worker cgroup's `memory.stat` stays near that limit. {% hint style="info" %} With `-XX:+PerfDisableSharedMem`, `jps`, `jstat`, and `jcmd -l` no longer list the worker JVM; `jcmd ` still works. {% endhint %} ## 3. Worker Network Configuration ### Binding the Worker to a Specific NIC To bind the worker to a specific local NIC (and its associated IP), set the bind device for each service (example uses `NIC1`): ```properties alluxio.worker.rpc.bind.device=NIC1 alluxio.worker.web.bind.device=NIC1 alluxio.worker.data.bind.device=NIC1 ```