> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/administration/managing-job-service.md).

# Job Service

If you've already used `job load` or `job free`, you've been interacting with the Job Service — it's the distributed scheduling engine that drives those operations. This page explains how it works internally, how to monitor jobs in flight, and how to configure it for high availability and production scale.

For operation-specific commands, see:

* [`job load`](https://documentation.alluxio.io/ee-ai-en/cache/cache-loading.md) — preloading data into the cache
* [`job free`](https://documentation.alluxio.io/ee-ai-en/cache/cache-eviction.md) — releasing cached data

## 1. Architecture

A **job** is a named, persistent, distributed operation submitted to Alluxio — for example, loading a dataset into cache or freeing space across all workers. Every job follows the same lifecycle:

**WAITING** → **RUNNING** → **SUCCEEDED** / **FAILED** / **STOPPED**

Once submitted, a job enters the waiting queue and is written to the Job Store before any scheduling happens. Whether that state survives a coordinator loss depends on the Job Store backend — see [Section 3](#id-3.-failure-recovery). The scheduler then claims the job, breaks it into per-worker tasks, and tracks completion. See [Section 2](#id-2.-listing-and-monitoring-jobs) for the full state reference.

The Job Service has three components that drive this lifecycle:

* **Coordinator** — a stateless scheduler that polls the job queue, claims jobs atomically, partitions them into file-batch tasks, and dispatches those tasks to workers. Multiple coordinators can run simultaneously; any instance can handle any job.
* **Job Store** — durable storage for job state. By default this is a local RocksDB instance on the coordinator node. In HA mode, all coordinators share an etcd cluster.
* **Workers** — execute the tasks dispatched by the coordinator: reading data from UFS and writing it to local cache (`load`), or evicting cached blocks (`free`).

{% hint style="warning" %}
The Coordinator is **not** the same as the Master in open-source Alluxio. Alluxio Enterprise Edition has replaced the Master with the Coordinator — a purpose-built stateless scheduler. There is no Master component in the enterprise edition. See [Key Components](https://documentation.alluxio.io/ee-ai-en/how-alluxio-works.md#key-components) for an overview.
{% endhint %}

## 2. Listing and Monitoring Jobs

### Job Identity

The default way to address a job is by its **(type, path)** pair — used throughout the CLI (`--path`, `--type`). Internally every job also has a UUID (visible in `job list` output), and some REST endpoints accept it directly.

### Listing Jobs

To see all jobs across the cluster:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job list --job-type ALL --job-state ALL
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio job list --job-type ALL --job-state ALL
```

{% endtab %}
{% endtabs %}

Use `--job-state` to filter by state and `--job-type` to filter by type (`LOAD`, `FREE`, `COPY`, `MOVE`). In HA mode, `--coordinator <address>` lists only jobs owned by a specific coordinator. Each entry in the output shows the job's `(type, path)` — use those values to query progress or stop the job.

### Querying a Specific Job

Once you have the `(type, path)` of a job from `job list`, query or control it:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# Query progress (load)
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <path> --progress

# Stop a running job
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path <path> --stop
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# Query progress (load)
bin/alluxio job load --path <path> --progress

# Stop a running job
bin/alluxio job load --path <path> --stop
```

{% endtab %}
{% endtabs %}

The same pattern applies to other job types: `alluxio job free --path <path> --progress`, etc.

### Job States

| State         | Description                                                                                                                                                                                                             |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **WAITING**   | Queued in the Job Store; not yet claimed by a coordinator.                                                                                                                                                              |
| **RUNNING**   | Coordinator is dispatching tasks to workers. Monitor with `--progress`.                                                                                                                                                 |
| **VERIFYING** | All tasks done; coordinator is running a verification pass (`--verify`).                                                                                                                                                |
| **SUCCEEDED** | All tasks completed successfully.                                                                                                                                                                                       |
| **FAILED**    | Error threshold exceeded. Inspect with `--progress --file-status FAILURE`.                                                                                                                                              |
| **STOPPED**   | Manually stopped via `--stop`. On both the RocksDB and the etcd job store, a stopped job is still listed as STOPPED after a coordinator restart and is removed after `alluxio.job.retention.time`, like a finished job. |

State transitions are atomic and at-most-once, so consistency is guaranteed even if multiple coordinators are active simultaneously.

## 3. Failure Recovery

Recovery behavior depends on the Job Store backend:

* **RocksDB (single coordinator, default):** Job state is written to RocksDB on the coordinator's local filesystem. If the RocksDB directory is on persistent storage (e.g., a PVC on Kubernetes — see [Production Setup](https://documentation.alluxio.io/ee-ai-en/administration/managing-alluxio.md#production-setup)), the state survives process crashes and pod restarts attached to the same volume. If the coordinator node is permanently lost or the metastore is backed by non-persistent storage, the queue and in-flight job state are lost along with it.
* **etcd (HA mode):** Job state lives in etcd, independent of any individual coordinator. A coordinator can be lost permanently and the remaining instances continue processing from the same shared queue.

### Automatic Recovery (HA Mode)

In HA mode, jobs actively managed by a crashed coordinator are recovered automatically by its peers:

* **Detection:** Active coordinators detect "stale" jobs in the **RUNNING** state that belong to an instance no longer sending heartbeats.
* **Re-queueing:** These orphaned jobs are reverted to **WAITING** so another healthy coordinator can pick them up.

In single-coordinator mode (RocksDB), there is no peer to perform detection — recovery happens when the coordinator process itself restarts and re-reads its local RocksDB.

### Worker Failure

Tasks running in a worker's thread pool are lost if the worker restarts mid-execution. The coordinator detects the stale tasks and retries each failed task. While a worker refuses connections (for example during a restart), retries to it back off exponentially (1, 2, 4, 8, 16 s, then every 30 s), so the 40 attempts per file last about 18 minutes and a normal restart does not fail any file. If retries are exhausted, the task is marked as failed and may eventually cause the job to transition to **FAILED** if the error threshold is exceeded.

### Manual Recovery (`job rerun`)

{% hint style="warning" %}
`job rerun` requires **etcd-based HA mode** (`alluxio.coordinator.job.meta.store.type=ETCD`). It will fail with `UnavailableException: Job ETCD is not enabled` in single-coordinator RocksDB mode. For RocksDB deployments, recovery happens automatically when the coordinator process restarts and re-reads local RocksDB.
{% endhint %}

In scenarios where a job gets stuck due to a complex failure (e.g., network partition or complete cluster restart), administrators can manually re-queue it:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# Rerun a specific load job that might be stuck
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job rerun --path /data/my-dataset --type load

# Rerun all jobs assigned to a specific (failed) coordinator
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job rerun --coordinator <failed_coordinator_host>
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# Rerun a specific load job that might be stuck
bin/alluxio job rerun --path /data/my-dataset --type load

# Rerun all jobs assigned to a specific (failed) coordinator
bin/alluxio job rerun --coordinator <failed_coordinator_host>
```

{% endtab %}
{% endtabs %}

## 4. High Availability

Even though the coordinator is never in the I/O critical path, a single coordinator backed by local RocksDB can still become a single point of failure. During a coordinator node upgrade — hardware replacement or a software update — `load` and `free` jobs cannot be submitted or make progress until the node comes back online.

To eliminate this, deploy multiple coordinator instances. Multiple schedulers sharing the same job queue require a distributed state store: this is where **etcd** comes in. All coordinators point to the same etcd cluster, so any instance can pick up and process jobs from the shared queue.

* **Stateless Coordinators:** Since coordinators do not hold local state, any instance can handle any job — there is no "leader" election for scheduling.
* **Fault Tolerance:** If one coordinator fails, the others continue processing the shared queue without interruption.
* **Failover:** A failed coordinator's in-flight jobs are detected as stale and re-queued within seconds by the remaining instances.
* **Load Balancing:** All coordinators consume from the same shared global job queue — work is naturally distributed. Polling intervals include random jitter to prevent thundering-herd bursts.

### Configuring HA

The key requirement is setting `alluxio.coordinator.job.meta.store.type=ETCD` on every coordinator node to switch job state storage from local RocksDB to the shared cluster.

{% tabs %}
{% tab title="Kubernetes (Operator)" %}
Set `coordinator.count` to deploy multiple replicas. The Operator provisions and manages etcd automatically.

```yaml
spec:
  coordinator:
    count: 3
  properties:
    alluxio.coordinator.job.meta.store.type: "ETCD"
```

```shell
kubectl apply -f alluxio-cluster.yaml
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}
Add to `alluxio-site.properties` on **every** coordinator node, then start each coordinator process:

```properties
alluxio.coordinator.job.meta.store.type=ETCD
```

```shell
bin/alluxio process start coordinator
```

{% endtab %}
{% endtabs %}

#### Optional: Dedicated etcd cluster for job scheduling

By default the coordinator stores job metadata in the main Alluxio etcd cluster (`alluxio.etcd.endpoints`). For workloads with heavy job-submission traffic, you can route job metadata to a **separate** etcd cluster to isolate it from service-discovery traffic:

```properties
# Defaults to alluxio.etcd.endpoints if unset
alluxio.coordinator.scheduler.load.job.etcd.endpoints=http://job-etcd1:2379,http://job-etcd2:2379
```

This applies to both Kubernetes and Docker / Bare-Metal deployments. On Kubernetes, add the property under `spec.properties` in the AlluxioCluster CR; on Docker / Bare-Metal, add it to `alluxio-site.properties` on every coordinator node.

### Verifying HA

After the coordinators come online, confirm that they are actually sharing the etcd-backed job queue.

**1. All coordinator replicas are running.**

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl -n <NAMESPACE> get pods -l app.kubernetes.io/component=coordinator
```

Expected: `coordinator.count` pods, all `Running` with `READY 1/1`.
{% endtab %}

{% tab title="Docker / Bare-Metal" %}
On each coordinator host:

```shell
docker ps --filter name=alluxio-coordinator
```

Expected: the container is `Up` on every host.
{% endtab %}
{% endtabs %}

**2. Coordinators share the same job queue.** Submit a job through one coordinator and query it from a different one — in HA mode both answer from the same etcd-backed queue.

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# Submit via coordinator-0
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job load --path /<mount>/<path> --submit

# Query from coordinator-1
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-1 -- \
  alluxio job load --path /<mount>/<path> --progress
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# From coordinator-1 host
bin/alluxio job load --path /<mount>/<path> --submit

# From coordinator-2 host
bin/alluxio job load --path /<mount>/<path> --progress
```

{% endtab %}
{% endtabs %}

Expected: both commands return the same job state. In single-coordinator (RocksDB) mode, the second query would report the job as not found.

**3. Failover test** (optional). Stop the coordinator currently driving a long-running job; a peer picks it up within `alluxio.coordinator.failure.detection.timeout` (default 15 s). Re-query `alluxio job load --path ... --progress` from a surviving coordinator and confirm the job advances past the failure point and eventually reaches `SUCCEEDED`.

## 5. UFS Load Governor

Every worker runs up to `alluxio.worker.job.load.executor.threads.max` (512) distributed-load UFS calls at once, whatever the store can take. With many workers sharing one bucket, the total can exceed what the store serves: the store throttles or queues, retries add to the pressure, and adding workers makes a load **slower** instead of faster. The load governor limits how many distributed-load UFS calls (directory listing, metadata lookups, page loads) each worker has in flight. Client reads and writes are not affected.

It works in one of two ways:

* **Adaptive**: each worker keeps a window per UFS that backs off when the store throttles, returns server errors, times out, or its latency rises well above normal, and grows back as it recovers. The window never exceeds `alluxio.worker.job.load.executor.threads.max`.
* **Manual**: every worker is pinned to a fixed number of concurrent load UFS calls, shared across all UFSes.

{% hint style="warning" %}
The adaptive governor is experimental. By default it runs in `SHADOW` mode, which observes and reports but never limits; turning on `ENFORCE` is an explicit operator decision. Known limitations: workers sharing a store that queues requests first-in first-out can estimate different normal latencies and settle at uneven windows; a load that mixes very small and very large files can be held at its initial window; and a window pushed to its minimum recovers slowly. If the adaptive window misbehaves, use a fixed manual limit instead.
{% endhint %}

### Choosing a Mode

The governor is used on demand. Keep the default unless the UFS is the bottleneck of your loads.

| Mode                          | When to use it                                                                                                                                                                                                                                             |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SHADOW` (default)            | Every cluster. Nothing is limited; the metrics show whether loads are pushing the UFS past what it can serve, and what window `ENFORCE` would use.                                                                                                         |
| `ENFORCE`                     | The UFS has limited capacity for this cluster's loads: many workers share one bucket, the store is self-hosted (MinIO, Ceph) or behind a gateway, the store returns many 503 / throttling errors during loads, or a load gets slower as workers are added. |
| Manual (`manual.permits` > 0) | You already know how much load concurrency the UFS can take, or the adaptive window does not suit the store (see the limitations above).                                                                                                                   |
| `OFF`                         | Removes the governor's bookkeeping entirely. Rarely needed: `SHADOW` already never blocks a call.                                                                                                                                                          |

### Settings

| Property                                                 | Default  | Changes at runtime                 | Description                                                                                                                                                                                                                                                     |
| -------------------------------------------------------- | -------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio.worker.job.load.governor.mode`                  | `SHADOW` | Yes                                | `OFF`: disabled. `SHADOW`: computes the window and exports metrics, never blocks a call. `ENFORCE`: load UFS calls wait for the window.                                                                                                                         |
| `alluxio.worker.job.load.governor.manual.permits`        | `0`      | Yes                                | A positive `N` limits each worker to `N` concurrent load UFS calls across all UFSes, whatever the mode; `0` leaves control to the mode. Setting the mode does not change it. A value at or above `alluxio.worker.job.load.executor.threads.max` limits nothing. |
| `alluxio.worker.job.load.governor.target.latency.factor` | `2.0`    | No (restart workers)               | The window stops growing when a round's median call latency exceeds this multiple of the estimated normal latency, and shrinks after three such rounds in a row. Lower is more conservative.                                                                    |
| `alluxio.worker.job.load.governor.floor.interval`        | `10sec`  | No (restart workers)               | Minimum progress when fully congested: at least one call per interval per worker, UFS, and call group (twice that for a group of metadata calls only).                                                                                                          |
| `alluxio.underfs.s3.congestion.domain`                   | `bucket` | No (mount option or site property) | Which S3 calls share one adaptive window. `bucket` suits AWS and S3-compatible public clouds, which throttle per bucket. Set `endpoint` for self-hosted stores and proxies (MinIO, Ceph, a gateway) whose capacity all buckets share.                           |

### Turn On Adaptive Limiting

1. **Observe first.** Leave the default `SHADOW` and run a representative large load. In the **Load Governor** row of the Grafana dashboard, **Shadow Mode Would-Block Rate** shows how many calls `ENFORCE` would have held back, and **Governor Window (permits)** shows the window it would use.
2. **Enable `ENFORCE` at runtime.** The coordinator stores the override in etcd and every worker applies it within about 10 seconds, with no restart:

   ```shell
   curl -sS 'http://<COORDINATOR_HOST>:19999/api/v1/conf' -X PUT \
     -H 'Content-Type: application/json' \
     --data '{"key":"alluxio.worker.job.load.governor.mode","value":"ENFORCE"}'
   ```
3. **Verify on the workers.** `alluxio_worker_load_governor_mode` (dashboard panel **Governor Mode**) reads `2` on every worker. Its values are `0` OFF, `1` SHADOW, `2` ENFORCE, `3` manual.
4. **Undo.** Set the mode back to `SHADOW` (or `OFF`) the same way. Calls already waiting for the window are released at once.

The value set with `PUT` is stored in etcd and still applies after workers restart, so you do not need to also add it to `alluxio-site.properties` or `spec.properties`.

### Set a Fixed Concurrency Manually

Use a manual limit when the adaptive window does not suit the store, or when you already know a good per-worker concurrency. It takes effect within about 10 seconds:

```shell
curl -sS 'http://<COORDINATOR_HOST>:19999/api/v1/conf' -X PUT \
  -H 'Content-Type: application/json' \
  --data '{"key":"alluxio.worker.job.load.governor.manual.permits","value":64}'
```

* `alluxio_worker_load_governor_mode` reads `3` and `alluxio_worker_load_governor_manual_permits` (panel **Manual Permits**) shows `N` on every worker.
* The limit is on concurrency per worker, not on request rate or bandwidth. Cluster-wide concurrency is `N` times the number of workers, so lower `N` when you add workers.
* Calls the adaptive path had already admitted or queued finish there, so a worker can briefly exceed `N` right after you set it.
* To hand control back to the mode, set the value to `0`.

In internal testing on AWS S3 with many workers loading small objects, a fixed value well below the 512-thread default cut S3 throttling errors and call latency without lowering throughput. The right value depends on the store, the object size, and the number of workers; start from the window the adaptive governor settles at (**Governor Window (permits)**).

### Things to Know About Runtime Overrides

* **Deleting an override does not revert running workers.** They keep the last value until restarted. To go back, `PUT` the value you want (for example `SHADOW` or `0`) instead of deleting the key.
* **A JVM system property wins over the runtime override.** If the key is set with `-D` in the worker's JVM options, a `PUT` has no effect on that worker. Values from `alluxio-site.properties` or `spec.properties` are overridden by the `PUT`.
* **Read the current override** with `curl -sS 'http://<COORDINATOR_HOST>:19999/api/v1/conf?key=alluxio.worker.job.load.governor.mode'`. Check the worker metrics above to confirm what each worker actually uses.
* During a rolling upgrade, set the keys after every worker runs a release that knows them; an older worker ignores them.

### Governor Metrics

All panels are in the **Load Governor** row of the default Grafana dashboard. Labels: `ufs` is the UFS (for S3, the bucket or endpoint); `group` is the set of calls sharing a window: `read_stat` (GET and HEAD) and `list` on S3, `read` and `stat_list` on other UFSes.

| Metric                                                                                     | What it tells you                                                                                                                         |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio_worker_load_governor_mode`                                                        | Mode in effect on the worker: `0` OFF, `1` SHADOW, `2` ENFORCE, `3` manual.                                                               |
| `alluxio_worker_load_governor_manual_permits`                                              | The manual limit in effect; `0` when none.                                                                                                |
| `alluxio_worker_load_governor_permits`                                                     | Current window per `ufs` and `group`.                                                                                                     |
| `alluxio_worker_load_governor_inflight` / `_utilization`                                   | Calls in flight in the last control round, and that count divided by the window.                                                          |
| `alluxio_worker_load_governor_md_total`                                                    | Window reductions by `signal`: `throttle`, `timeout`, `distress` (server errors), `latency`.                                              |
| `alluxio_worker_load_governor_round_latency_ms` / `_base_latency_ms` / `_latency_brake_ms` | Median call latency of the last round, the estimated normal latency, and the line (factor × normal) above which the window stops growing. |
| `alluxio_worker_load_governor_rounds_total`                                                | Control rounds by `decision` (for example `grow`, `hold`, `reduce`, `shrink_latency`), which explains why the window moved.               |
| `alluxio_worker_load_governor_acquire_wait_ms`                                             | Time calls waited for the window in `ENFORCE` or manual mode.                                                                             |
| `alluxio_worker_load_governor_shadow_would_block_total`                                    | Calls `SHADOW` mode would have held back.                                                                                                 |
| `alluxio_ufs_business_retries_total` / `alluxio_worker_load_inplace_retries_total`         | Retries absorbed by the object-store retry layer and by the worker's in-place retry of load calls.                                        |

Retries inside the S3 SDK are not visible to Alluxio. With the default S3 client, most throttling responses are retried inside the SDK and show up only as higher call latency, which is why the governor also reacts to latency. Compare with the store's own metrics (for S3, the 5xx count in CloudWatch) when tuning.

## 6. Configuration and Tuning

You can fine-tune the Job Service behavior for your specific workload and etcd capacity.

| Property                                                     | Default          | Description                                                                                                                                                                                       |
| ------------------------------------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio.coordinator.dora.load.job.failed.file.retained.max` | `100000`         | Maximum number of failed file paths a running load job keeps in memory and writes to its failed-file list. Failures beyond it are counted but not listed, and the job ends FAILED.                |
| `alluxio.coordinator.failure.detection.timeout`              | `15s`            | Time before a silent coordinator is considered failed and its jobs reclaimed by peers.                                                                                                            |
| `alluxio.coordinator.scheduler.etcd.max.retries`             | `2`              | Max retries for etcd operations before giving up.                                                                                                                                                 |
| `alluxio.coordinator.scheduler.etcd.read.timeout`            | `10s`            | Timeout for reading from etcd.                                                                                                                                                                    |
| `alluxio.coordinator.scheduler.etcd.write.timeout`           | `15s`            | Timeout for writing to etcd.                                                                                                                                                                      |
| `alluxio.coordinator.scheduler.interval.time`                | `2s`             | Scheduling interval. A shorter interval allows faster scheduling of small jobs but increases CPU overhead.                                                                                        |
| `alluxio.coordinator.scheduler.job.etcd.path`                | `/alluxio/jobs/` | The prefix directory in etcd used for storing job metadata.                                                                                                                                       |
| `alluxio.coordinator.scheduler.job.etcd.poll.interval`       | `1s`             | How often coordinators check etcd for new waiting jobs. Lower values mean faster response but higher load on etcd.                                                                                |
| `alluxio.coordinator.scheduler.max.tasks.per.worker`         | `3`              | Maximum number of tasks each coordinator can schedule simultaneously on the same worker. Increasing this raises concurrency but risks creating a backlog on the worker.                           |
| `alluxio.coordinator.scheduler.running.job.capacity`         | `100`            | Maximum number of jobs the scheduler can run concurrently. Jobs beyond this limit remain in the waiting queue.                                                                                    |
| `alluxio.coordinator.scheduler.waiting.job.capacity`         | `100000`         | Maximum number of jobs that can wait in the scheduler queue. Submissions beyond this limit are rejected.                                                                                          |
| `alluxio.job.batch.size`                                     | `600`            | Number of files per task dispatched to a worker for `load`. The default works well across file sizes; override per job via CLI (`--batch-size`) only if a specific workload benefits.             |
| `alluxio.job.free.rate.limit`                                | `20000`          | Per-worker release rate for `free`, in files per second, shared across all running free jobs on a worker; `0` means unlimited. Override per job via CLI (`--rate-limit`).                         |
| `alluxio.job.cleanup.interval`                               | `1H`             | Interval for periodic cleanup of finished jobs from the job store.                                                                                                                                |
| `alluxio.job.retention.time`                                 | `7d`             | Job history retention time. Load job file lists on the coordinator are deleted after the same period.                                                                                             |
| `alluxio.worker.job.load.executor.threads.max`               | `512`            | Maximum number of threads in the worker's job executor pool for parallel UFS data loading. Also the upper bound of the load governor's window; see [UFS Load Governor](#id-5.-ufs-load-governor). |
| `alluxio.worker.load.file.partition.size`                    | `64MiB`          | Shard size for splitting large files during loading. Smaller shards increase concurrency but may create more overhead.                                                                            |

### Small File Optimization

The default batch size (`alluxio.job.batch.size = 600`) is tuned to keep workers busy across a wide range of file sizes, so you normally do not need to change it. You can still override it per job with `--batch-size` if a specific workload benefits:

```shell
bin/alluxio job load --path s3://bucket/small-files/ --submit --batch-size 2000
```

In practice, small-file load throughput is typically limited by the **S3 request rate (IOPS)** rather than Alluxio's internal concurrency. Once the S3 IOPS ceiling is reached, increasing `alluxio.worker.job.load.executor.threads.max` beyond its default of 512 or adding more workers will not improve throughput — the bottleneck has shifted to the object store. Past that point, more concurrency makes the store throttle and can slow the load down; see [UFS Load Governor](#id-5.-ufs-load-governor).

Large files are split internally into `alluxio.worker.load.file.partition.size` (64 MiB) shards, so a single task can already generate multiple concurrent subtasks; the default batch size accounts for this.

## 7. Metrics

{% hint style="info" %}
These metrics are low-level indicators of internal scheduler operations. For general cluster health and job progress, refer to the default Grafana dashboard.
{% endhint %}

| Metric                              | Description                                                                   |
| ----------------------------------- | ----------------------------------------------------------------------------- |
| `alluxio_scheduler_etcd_poll`       | Counter of poll operations to etcd.                                           |
| `alluxio_scheduler_etcd_claim`      | Counter of jobs successfully claimed from the queue.                          |
| `alluxio_scheduler_etcd_reclaim`    | Counter of jobs reclaimed/recovered from failed coordinators.                 |
| `alluxio_scheduler_etcd_latency_ms` | Latency distribution of etcd operations performed by the scheduler.           |
| `alluxio_scheduler_job_access_etcd` | Counter of various job-related access types (list, get, transaction) to etcd. |

## See Also

* [Cache Loading](https://documentation.alluxio.io/ee-ai-en/cache/cache-loading.md) — job submission, monitoring, and free operations
* [Installing on Kubernetes](https://documentation.alluxio.io/ee-ai-en/start/installing-on-kubernetes.md) — ETCD configuration and multi-coordinator deployment
* [Cluster and Membership Management](https://documentation.alluxio.io/ee-ai-en/administration/managing-alluxio.md) — resource tuning, node pinning, persistent metastore
* [Monitoring](https://documentation.alluxio.io/ee-ai-en/administration/monitoring-alluxio.md) — Grafana dashboards, alert rules, metrics reference
