Ctrlk
ProductsLanguageHome
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Worker Configuration

This guide covers configuration for individual Alluxio workers: storage backends, capacity sizing, resource limits, JVM tuning, and network binding. For hash-ring-related worker operations (adding, removing, restarting, identity persistence), see Hash Ring and Worker Lifecycle. For cluster-wide operations (scaling, upgrades, coordinator, UFS, multi-tenancy), see Cluster Management.

1. Worker Storage

Each worker caches data in a local page store. This section covers the storage backend choice, capacity sizing, and disk layout.

Configuring Page Store Location

The Operator supports two page store backends.

Default (hostPath): The worker writes cache to the node's filesystem at /mnt/alluxio/pagestore.

spec:
  worker:
    pagestore:
      # Defaults to hostPath: /mnt/alluxio/pagestore on the node's filesystem.
      size: 100Gi
      reservedSize: 10Gi

PVC-backed: To persist worker cache data across pod restarts or rescheduling, specify a PersistentVolumeClaim (PVC) for the page store.

spec:
  worker:
    pagestore:
      type: persistentVolumeClaim
      storageClass: ""    # defaults to "standard"; empty string = static binding
      size: 100Gi
      reservedSize: 10Gi

Set the page store directory and size via Alluxio properties. On Docker, pass them through ALLUXIO_JAVA_OPTS and mount the host directory as a volume so the cache survives container recreation:

docker run ... \
  -v /data/alluxio-cache:/data/alluxio-cache \
  -e ALLUXIO_JAVA_OPTS=" \
    -Dalluxio.worker.page.store.dirs=/data/alluxio-cache \
    -Dalluxio.worker.page.store.sizes=<CACHE_SIZE>" \
  alluxio/alluxio-enterprise worker

On bare-metal, set the same properties in conf/alluxio-site.properties:

alluxio.worker.page.store.dirs=/data/alluxio-cache
alluxio.worker.page.store.sizes=<CACHE_SIZE>

Sizing the Page Store

  • size: Per-worker cache capacity. Must not exceed available disk on the worker node.

  • reservedSize: Space reserved for internal operations (temporary page writes, metadata caching). Set to ~10% of size, typically 10–100 GiB.

  • Ensure size + reservedSize ≤ available disk space.

Cloud providers advertise disk size in GB (base-10), while Kubernetes Gi is base-2. A "1000 GB" EBS volume provides ~931 GiB. Set size to ~90% of actual available space (check with df -h <page-store-path>) to leave room for filesystem overhead and reservedSize. A too-large size causes workers to crash with quota (NNN) exceeds the total disk space.

Multi-Disk Configuration

For nodes with multiple data disks, configure the page store to span all of them — this distributes page I/O across disks and increases aggregate throughput. Use comma-separated paths and sizes:

Each directory must map to a separate physical disk — colocating multiple page store directories on the same disk provides no benefit.

An alternative is to use RAID 0 at the OS level to present multiple disks as a single logical volume, then configure a single hostPath. This simplifies the Alluxio configuration but couples the lifetime of all disks — a single disk failure loses the entire array.

Heterogeneous Workers

The workerGroups mechanism described in this section is specific to the Kubernetes Operator. On Docker/bare-metal deployments, heterogeneous workers are achieved by running each worker with its own alluxio-site.properties file that specifies the per-node page store paths and sizes.

When a cluster has workers with different disk specifications (e.g., one group with a 1 TB disk, another with two 800 GB disks), use workerGroups to define distinct configurations per group.

Step 1: Group and label nodes:

Step 2: Define worker groups and enable capacity-aware hash ring.

For heterogeneous clusters, set alluxio.user.worker.selection.policy.consistent.hash.provider.impl to CAPACITY so workers with more storage receive a proportionally larger share of data. For details on this property, see Optimizing for Heterogeneous Workers.

While this provides flexibility, it is crucial to ensure consistency within each worker group. Misconfigurations can lead to unexpected errors.

2. Resource and JVM Tuning

Configure per-component resource limits and JVM options in alluxio-cluster.yaml:

On Docker, set JVM options through ALLUXIO_JAVA_OPTS and the container resource limits via --memory and --cpus:

On bare-metal, set JVM options in conf/alluxio-env.sh:

Memory Limit Formula

For the worker config above (-Xmx22g, -XX:MaxDirectMemorySize=10g): minimum limit is 22 + 10 + 2 = 34 GiB, set to 36 GiB in the example.

If -XX:MaxDirectMemorySize is omitted, the JVM defaults it to the same value as -Xmx, so the container limit typically needs to be 2.5× -Xmx or more.

Diagnosing OOM

If a worker is killed due to OOM (exit code 137), confirm the cause using the commands below.

Symptom
Root Cause
Fix

Exit Code 137, no Java error

Container limit exceeded — killed by Linux OOM killer

Increase resources.limits.memory

java.lang.OutOfMemoryError: Java heap space

-Xmx too small

Increase -Xmx and raise container limit accordingly

java.lang.OutOfMemoryError: Direct buffer memory

-XX:MaxDirectMemorySize too small

Increase -XX:MaxDirectMemorySize and raise container limit accordingly

Diagnosing Stalls

If peers log redirect or connect timeouts to a worker and reads fall back to the UFS while the worker pod stays READY, the worker JVM is probably stopping for seconds at a time, either for garbage collection (GC) or at a safepoint, where the JVM stops all Java threads for some internal operations. Check for that before tuning the network: rising TCP listen-queue drops on the node are a result of the pauses, not their cause.

1. Confirm the pauses. The worker logs every pause longer than 1 second as JVM paused <N>ms (at WARN above 10 seconds). The line after it says whether a GC ran during the pause:

GC list: means a GC ran during the pause; No GCs detected means the JVM stopped for something else. GC metrics alone cannot tell these apart, because they do not count non-GC pauses.

2. Collect a GC and safepoint log, over a few hours: a short capture can single out the wrong cause. Add the option to the worker JVM options, which takes effect on the next restart:

Or turn the same log on in a running worker, without a restart. The setting is lost when the worker restarts.

On Kubernetes, copy the log out after a few hours with kubectl cp alx-ns/<worker-pod-name>:/opt/alluxio/logs/gc.log gc.log.

3. Match the log to a cause. Each GC appears as a group of lines with the same GC(<n>) number: the pause and its total time, the time of each phase, and the CPU time its threads used. Each safepoint appears as one Safepoint line, with its times in nanoseconds. A stall looks like this:

Here a 7-second young GC spent almost all of its time in Pre Evacuate Collection Set, and a Cleanup safepoint took 4.8 seconds to reach. Start from the rows that match what step 1 showed.

Step 1 shows
Symptom in the GC and safepoint log
Root Cause
Fix

GC list

Pause Full or To-space exhausted, or long Pause Young with most of the time in Evacuate Collection Set

Heap too small for the live data

Increase -Xmx and raise container limit accordingly

GC list

Long GC pause whose User CPU time is far below Real

GC threads held off the CPU by the container's CPU limit (nr_throttled rising in the cgroup's cpu.stat)

Raise the CPU limit

GC list or No GCs detected

Long Pause Young with most of the time in Pre Evacuate Collection Set, or Safepoint lines whose Reaching safepoint takes seconds

Threads writing the JVM's mmapped /tmp/hsperfdata_<user>/<pid> file block on the container's dirty-page limit

Add -XX:+PerfDisableSharedMem to the worker JVM options

No GCs detected

Seconds-long Safepoint "FindDeadlocks" lines at every metrics scrape

alluxio.metrics.jvm.deadlock.detection.enabled=true

Leave the property at its default, false

For the hsperfdata case, the dirty-page limit is the node's vm.dirty_bytes or vm.dirty_ratio, scaled down to each container's memory cgroup; an absolute vm.dirty_bytes can leave a container only a few MB. To confirm, check that file_dirty in the worker cgroup's memory.stat stays near that limit.

With -XX:+PerfDisableSharedMem, jps, jstat, and jcmd -l no longer list the worker JVM; jcmd <pid> still works.

3. Worker Network Configuration

Binding the Worker to a Specific NIC

To bind the worker to a specific local NIC (and its associated IP), set the bind device for each service (example uses NIC1):

Last updated