> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/cache/loading-data-into-the-cache.md). # Cache Loading Alluxio populates its cache in two ways: **passively** on first read (automatic, no setup) and **actively** via the `job load` command (explicit preload before your job runs). ## Prerequisites * A running Alluxio cluster with at least one worker * At least one UFS mount configured (`alluxio mount list` to verify) {% hint style="info" %} Alluxio will automatically evict cached data to make room for new data according to the configured eviction policy. You do not need to pre-clear space before submitting a load job. {% endhint %} ## Passive Caching On every cache miss, Alluxio fetches the file from UFS and writes it into the worker cache while streaming it to the application. No configuration needed — subsequent reads are served from cache. This is the default behavior. Use active preloading when you cannot afford the first-read latency. ## Active Preloading with `job load` `job load` submits a distributed load job: the coordinator distributes work across all workers, each pulling its assigned files from UFS directly. For scheduling internals, HA, and advanced tuning, see [Job Service](/ee-ai-en/ai-3.8-15.1.x/administration/managing-job-service.md). ### Quick Reference There are three **mutually exclusive** ways to specify what to load (pick one); the table also lists the common follow-up tasks: | Scenario / Task | How | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Load a directory or a whole dataset (recursively) | `--path` (below) | | An explicit file list stored on UFS | `--index-file` (see *Loading from an Index File*) | | A file list on the client (e.g., script-generated), without uploading it to UFS | `--local-index-file` (see *Loading from an Index File*) | | Fill gaps after some files failed | re-run with `--skip-if-exists` (see *Key Flags* / *Failure Modes*) | | Keep the cache in sync when the source is overwritten in place | `--load-policy IF_CHANGED` (see *Incremental Load for Mutable Data*) | | Refresh directory listings after files were added/removed behind Alluxio's back | `--with-index-service` / `--only-index-service` (see *Rebuilding the Listing Index*) | | Change a parameter and re-run after submitting | add `--overwrite` (see *Changing a Running Job's Parameters*) | | Very large dataset (hundreds of thousands of files or more) | split into multiple jobs (see *Splitting Very Large Datasets*) | The examples below use `--path`; the two index-file forms are covered under *Loading from an Index File*. ### Before You Submit Three quick checks prevent most "everything failed" incidents: 1. **Mount-point path spelling** — the target must be under a mounted UFS. A misspelled bucket makes every file fail instantly and the job FAIL within seconds with zero bytes loaded. Verify with `alluxio mount list`. 2. **A clean file list** — entries that do not exist are each charged as one failed file (retried up to the cap, then given up); the rest still load, but the job ends FAILED. Clean the list beforehand when you can. 3. **Re-submits are idempotent** — re-submitting the same path/list while a job is still running does not create a new job or change its parameters; it merges into the existing one. To change parameters, use `--overwrite` (see *Changing a Running Job's Parameters*). ### Submit and Monitor `--path` accepts either a UFS path (e.g. `s3://my-bucket/dataset/`) or an Alluxio virtual path (e.g. `/mnt/dataset/`). See the [CLI reference](/ee-ai-en/ai-3.8-15.1.x/reference/user-cli.md#job-load) for details. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # Submit (returns immediately) kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit # Monitor progress kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --progress ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # Submit (returns immediately) bin/alluxio job load --path --submit # Monitor progress bin/alluxio job load --path --progress ``` {% endtab %} {% endtabs %} Example progress output: ```console Progress for loading path 's3://my-bucket/dataset/': Settings: replicas: unset batch-size: 600 verify: false metadata-only: false quota-check: false Time start: 2026-04-15T22:05:01 Time finished: 2026-04-15T22:05:08 Time Elapsed: 7s Job State: SUCCEEDED Inodes Scanned: 1000 Non Empty File Copies Loaded: 1000 Bytes Scanned: 125.00MiB Bytes Loaded: 125.00MiB Throughput: 17.86MiB/s File Failure rate: 0.00% Subtask Failure rate: 0.00% Files Failed: 0 Subtask Retry rate: 0.00% Subtasks on Retry Dead Letter Queue: 0 ``` ### Submitting via the REST API Every `job load` operation is also available on the coordinator's REST endpoint (`POST /api/v1/load` on port `19999`), which is convenient for programmatic submission. The request body fields mirror the CLI flags: * `path` — a single directory or file (like `--path`) * `index` — a UFS index file (like `--index-file`) * `paths` + `alias` — an inline path list (like `--local-index-file`); `alias` is a caller-chosen name that identifies the job for later progress/stop queries * `isOverWrite` — top-level, like `--overwrite` * `options` — a nested object: `skipIfExists`, `loadPolicy` (`"IF_CHANGED"`), `verify`, `batchSize`, `fileFilterRegex`, `indexServiceMode` (`"withLoad"` / `"indexOnly"`, see *Rebuilding the Listing Index*) ```shell # Submit a directory load curl -X POST http://:19999/api/v1/load \ -H 'Content-Type: application/json' \ -d '{"path": "s3://my-bucket/dataset/", "options": {"skipIfExists": true}}' # Submit an inline path list (requires an alias) curl -X POST http://:19999/api/v1/load \ -H 'Content-Type: application/json' \ -d '{"paths": ["s3://my-bucket/a.parquet", "s3://my-bucket/day=1/"], "alias": "warmup-1"}' # Query progress — by path, or by alias for an inline-list job curl -s 'http://:19999/api/v1/load?target=warmup-1' ``` For the full request/response schema, error codes, and the stop/file-list endpoints, see the [REST API reference](/ee-ai-en/ai-3.8-15.1.x/reference/rest-api.md). ### Stop a Running Job {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --stop ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --stop ``` {% endtab %} {% endtabs %} A stopped job can be resumed by submitting it again with `--submit`. Already-cached files will be skipped if `--skip-if-exists` is included. ### Key Flags | Flag | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `--submit` | Submit the job asynchronously (returns immediately) | | `--progress` | Show progress of a submitted job | | `--stop` | Stop a running job | | `--verify` | After load completes, re-check every file and reload any that are not fully cached. | | `--replicas ` | Load `n` replicas per file (default: 1); useful for high-concurrency reads | | `--skip-if-exists` | Skip files that are already fully cached (safe to re-run a load job) | | `--load-policy IF_CHANGED` | Re-check each cached file against UFS metadata; reload only files whose content has changed. Use for incremental sync of mutable datasets. | | `--metadata-only` | Load file metadata without caching file data | | `--batch-size ` | Number of files per batch per worker. Default: 600. The default works well across file sizes; you normally do not need to change it. | | `--partial-listing` | Start loading before the full directory listing completes; useful for very large directories | | `--index-file ` | Load a specific list of files defined in a UFS index file (one path per line) | | `--local-index-file ` | Like `--index-file`, but the manifest is read from the client's local filesystem and sent with the request — no UFS upload needed | | `--overwrite` | Terminate an existing job for the same path and submit a fresh one with new parameters (see *Changing a Running Job's Parameters*) | | `--with-index-service` | Rebuild the listing index for every directory in the loaded tree, concurrently with the data load (see *Rebuilding the Listing Index*) | | `--only-index-service` | Only rebuild the listing index — no data or metadata is loaded (see *Rebuilding the Listing Index*) | For the full flag reference, see [`job load` CLI documentation](/ee-ai-en/ai-3.8-15.1.x/reference/user-cli.md#job-load). ### Loading from an Index File For selective loading or when the directory tree is too large to traverse upfront: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --index-file s3://my-bucket/load-manifest.txt --submit ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --index-file s3://my-bucket/load-manifest.txt --submit ``` {% endtab %} {% endtabs %} Index file format — one UFS path per line, lines starting with `#` are comments: ``` s3://my-bucket/dataset/train/ s3://my-bucket/dataset/val/file.parquet # s3://my-bucket/dataset/test/ <- skipped ``` Each line is one of two kinds: * A **file** (no trailing `/`) is dispatched to a worker directly — the coordinator does no listing for it, which is the lowest-overhead form. * A **directory** (trailing `/`) is enumerated recursively by the coordinator. If a listed entry does not exist, that single line is counted as one failed file and the remaining lines continue (see the Failure Modes section). To submit a manifest that lives on the **client's local filesystem** (e.g., one generated by a script), use `--local-index-file ` instead — the client reads the file and sends it with the request, so it does not need to be uploaded to UFS first. The file format is identical. For very long manifests (hundreds of thousands of lines), prefer a UFS `--index-file`: a local manifest is sent in a single request, and a very large one can exceed the request-size limit. ### Incremental Load for Mutable Data When the underlying dataset changes periodically (e.g., daily model checkpoints, updated training splits), use `--load-policy IF_CHANGED` to sync only the files that have changed since the last load: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit --load-policy IF_CHANGED ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --submit --load-policy IF_CHANGED ``` {% endtab %} {% endtabs %} `--load-policy IF_CHANGED` re-checks UFS metadata for each **already-cached** file and reloads it only if the content has changed. Files that are not yet cached are loaded unconditionally. This makes it the right choice for periodic sync of a mutable dataset: new files get cached, changed files get refreshed, and unchanged files are skipped. | Flag | Cached files | Uncached files | | -------------------------- | ---------------------- | -------------- | | `--submit` (no flags) | Reload unconditionally | Load | | `--skip-if-exists` | Skip | Load | | `--load-policy IF_CHANGED` | Reload only if changed | Load | ### Rebuilding the Listing Index (Index Service) *Available since 3.8-15.1.18.* The **index service** is Alluxio's distributed cache for directory listings: once a directory's listing is indexed, `ls` (CLI and FUSE) is served from the index instead of listing the UFS on every call, which is what keeps listings of very large directories fast. See [Metadata Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/metadata-listing.md) for how it works and how to enable it. When files are added or removed in the UFS **behind Alluxio's back** (e.g., another pipeline writes directly to S3), a cached directory listing does not notice: `ls` on that directory keeps returning the stale list until the listing cache is freed or expires. Two `job load` modes rebuild the listing index as part of the job, so the listings served to `ls` (CLI and FUSE) match the UFS again: * `--with-index-service` — load data/metadata as usual **and** rebuild the listing index of every directory in the loaded tree. The index rebuild runs *concurrently* with the data load, so for typical loads it adds little or no wall time. * `--only-index-service` — rebuild the listing index only; **no data or metadata is loaded** (`Bytes Loaded: 0B`). This is the lightweight way to refresh listings after an out-of-band change when you do not need the new data cached yet. The two flags are mutually exclusive. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # Refresh listings only (no data movement) kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit --only-index-service # Load new data and refresh listings in one job kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit --with-index-service ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # Refresh listings only (no data movement) bin/alluxio job load --path --submit --only-index-service # Load new data and refresh listings in one job bin/alluxio job load --path --submit --with-index-service ``` {% endtab %} {% endtabs %} Progress reporting gains an extra line when either mode is active: ``` Settings: ... index-service: with-load Index Directories: 5 indexed / 5 scanned (failed: 0) ``` A directory that fails to index does **not** fail the job and is not counted in the file failure rate — the job reports `SUCCEEDED` with a note listing how many directories failed; re-run to retry them. Index rebuild failures are also exposed as the coordinator metric `alluxio_distributed_load_index_dirs_total{status="failed"}`. **Consistency after out-of-band UFS changes** — what `--with-index-service` guarantees for a directory it rebuilds: | Out-of-band change | Result after the load | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | File **added** | Listing, metadata, and data are all consistent with UFS (any flag combination) | | File **overwritten** | Consistent by default. With `--skip-if-exists`, the listing is refreshed but cached data/metadata keep the old version — add `--load-policy IF_CHANGED` to refresh all three. | | File **deleted** | The file disappears from its parent's listing. Its already-cached metadata/data are not proactively removed — direct `getStatus`/reads of the old path may still hit cache until TTL/eviction or a `job free`. | To drop a previously built listing index (e.g., before measuring a cold load), use `job free --path --index-service --submit`. Two coordinator properties bound the index work per job: * `alluxio.coordinator.dora.load.job.index.list.concurrency` (default `4`) — how many directories one job indexes concurrently. Each in-flight directory performs one single-level UFS listing; raise it to speed up indexing when the UFS has headroom. * `alluxio.coordinator.dora.load.job.index.max.dirs` (default `10000000`) — cap on directories collected per job (\~160 bytes of coordinator memory per directory); directories beyond the cap are counted as failed to index. The rebuild cost scales with the number of entries listed, not bytes loaded: hidden behind any reasonably long data load it is effectively free, while an index-only job on a single directory with millions of entries takes minutes (bounded by UFS listing speed). ### Changing a Running Job's Parameters Re-submitting the same path while a job is still running is idempotent — the running job keeps its original parameters and the newly supplied ones are ignored (the CLI reports `Load already running ... Other params will remain the same`). If you submitted a job and then need to change a submit-time parameter, you do not need to `--stop` and resubmit: add `--overwrite` to terminate the existing job and submit a fresh one with the new parameters in one step. The example below raises `--batch-size` — one of the few cases where overriding the default helps, e.g. a small-file workload where workers sit mostly idle. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit --overwrite --batch-size 2000 --skip-if-exists ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --submit --overwrite --batch-size 2000 --skip-if-exists ``` {% endtab %} {% endtabs %} The old job is terminated and marked `FAILED` (reason: `will be overwritten`) — this is expected. `--overwrite` **does not evict already-cached data** (it is not `job free`), so combine it with `--skip-if-exists`: the new job re-enumerates every file, but the worker skips those already fully cached and reloads only the remainder. ### Splitting Very Large Datasets A single job over hundreds of thousands of files works, but splitting into several medium-sized jobs is usually a better experience: a smaller failure blast radius (one job's failure affects only its slice) and faster gap-filling re-runs. Split by directory (one job per partition), or split a manifest into groups (one `--local-index-file` per group). This is about failure isolation and gap-filling granularity, not concurrency — the coordinator caps the number of simultaneously running jobs, so a handful to a dozen parallel jobs is the right scale. ## Integrating with ML Training A typical workflow: load data → verify → run training. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # 1. Submit load kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path s3://my-bucket/dataset/ --submit --verify # 2. Poll until SUCCEEDED kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path s3://my-bucket/dataset/ --progress # Repeat until "Job State: SUCCEEDED", then launch training pods ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # 1. Submit load bin/alluxio job load --path s3://my-bucket/dataset/ --submit --verify # 2. Wait until SUCCEEDED bin/alluxio job load --path s3://my-bucket/dataset/ --progress # Repeat until "Job State: SUCCEEDED" # 3. Start training python train.py --data /mnt/alluxio/fuse/dataset/ ``` {% endtab %} {% endtabs %} {% hint style="info" %} **Near-100% cache coverage:** For critical datasets, run a second pass with `--skip-if-exists` after the first job reaches `SUCCEEDED`. In rare cases — transient worker failures or hash ring boundary timing — a single pass may miss a small fraction of files. A second pass fills those gaps without re-loading already-cached data: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --submit --skip-if-exists ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --submit --skip-if-exists ``` {% endtab %} {% endtabs %} {% endhint %} ## Failure Modes `job load` is a best-effort, long-running task; it does **not** guarantee 100% success and is not all-or-nothing. On large datasets, occasional per-file failures are normal (transient object-store timeouts, a worker restarting). A final `Job State: FAILED` means *partially failed* — successfully loaded files remain cached; it does not mean nothing was loaded. Each failed file is retried a bounded number of times (default 40, `alluxio.coordinator.dora.load.job.subtask.max.retry.attempts`) before being counted as failed, so failures do not retry forever or wedge the job. The standard recovery is to re-run with `--skip-if-exists`. **`Job State: FAILED` with `Files Failed > 0`** Check the file-level failure list: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --progress --file-status FAILURE ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --progress --file-status FAILURE ``` {% endtab %} {% endtabs %} Common causes: UFS access errors, network timeouts, or missing credentials. Fix the underlying issue, then resubmit with `--skip-if-exists` to avoid re-loading already-cached files (forgetting `--skip-if-exists` makes the re-run reload everything, which can take a long time). The `Failed files saved to: ` line at the top of the progress report points to a complete failed-file list written on the coordinator — use it to build a gap-filling index file containing only the failed entries; `--progress --format JSON --verbose` gives structured detail and recent failure-reason samples. **`Job State: FAILED` immediately after submit** Run `--progress --verbose` for detail: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path --progress --verbose ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio job load --path --progress --verbose ``` {% endtab %} {% endtabs %} Often caused by: path not found in mount table (verify with `alluxio mount list`), or insufficient cache quota. **Load succeeds but reads still go to UFS** Verify that specific files are actually cached: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio fs check-cached ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio fs check-cached ``` {% endtab %} {% endtabs %} If files show as uncached after a successful load, data may have been evicted. Check cache capacity and eviction settings — see [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/removing-data-from-the-cache.md). For cluster-wide cache hit rate, see [Monitoring](/ee-ai-en/ai-3.8-15.1.x/administration/monitoring-alluxio.md). `check-cached` operates on a directory or a manifest (pass `--index-file ` to check a specific list); it does not accept a single file path. To verify one individual file, query a worker directly and check that `mInAlluxioPercentage` is `100`: ```shell curl -s 'http://:28080/v1/info?path=' ``` If the path contains `=` — e.g., Hive-style partition directories like `date=2026-07-01` — encode it as `%253D`; otherwise the path is truncated and the query returns 404. Troubleshooting quick reference: | Symptom | What to do | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Ends FAILED, but most files read fine and fast | Normal partial failure; check `Files Failed` and the failure list, re-run with `--skip-if-exists` to fill the gaps | | FAILED within seconds of submit, zero bytes loaded | Target is likely not under a mount point (misspelled bucket); verify the mounts and path | | The list contains files that do not exist | Each is charged as one failure; the rest load normally; final state FAILED — cross-check with the failure list | | A re-run takes a long time | You forgot `--skip-if-exists` (reloads everything); or gap-fill precisely from the failure list | | Source files updated but reads still return old content | Re-warm the path with `--load-policy IF_CHANGED` | | Confirm whether a specific file is cached | Single file: worker `/v1/info`; batch: `check-cached --index-file` (see above) | ## Retention of Historical Jobs Completed job records are kept for a configurable period. The default is 7 days. To adjust: ```properties # retain completed job records for 3 days (default: 7d) alluxio.job.retention.time=3d ``` ## Related * [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/removing-data-from-the-cache.md) — manual `job free`, version update patterns, and automatic eviction policies * [Job Service](/ee-ai-en/ai-3.8-15.1.x/administration/managing-job-service.md) — `job list`, job states, coordinator HA, failure recovery, and configuration tuning * [Multiple Replicas](/ee-ai-en/ai-3.8-15.1.x/high-availability/multiple-replicas.md) — load multiple copies per file for fault tolerance * [`job load` CLI Reference](/ee-ai-en/ai-3.8-15.1.x/reference/user-cli.md#job-load)