Ctrlk
ProductsHome
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Cache Loading

Alluxio populates its cache in two ways: passively on first read (automatic, no setup) and actively via the job load command (explicit preload before your job runs).

Prerequisites

  • A running Alluxio cluster with at least one worker

  • At least one UFS mount configured (alluxio mount list to verify)

Alluxio will automatically evict cached data to make room for new data according to the configured eviction policy. You do not need to pre-clear space before submitting a load job.

Passive Caching

On every cache miss, Alluxio fetches the file from UFS and writes it into the worker cache while streaming it to the application. No configuration needed — subsequent reads are served from cache.

This is the default behavior. Use active preloading when you cannot afford the first-read latency.

Active Preloading with job load

job load submits a distributed load job: the coordinator distributes work across all workers, each pulling its assigned files from UFS directly. For scheduling internals, HA, and advanced tuning, see Job Service.

Quick Reference

There are three mutually exclusive ways to specify what to load (pick one); the table also lists the common follow-up tasks:

Scenario / Task
How

Load a directory or a whole dataset (recursively)

--path (below)

An explicit file list stored on UFS

--index-file (see Loading from an Index File)

A file list on the client (e.g., script-generated), without uploading it to UFS

--local-index-file (see Loading from an Index File)

Fill gaps after some files failed

re-run with --skip-if-exists (see Key Flags / Failure Modes)

Refresh the cache after files were overwritten in place

--metadata-only (see Cache Invalidation)

Re-load only the changed files of a mutable dataset

--load-policy IF_CHANGED (see Incremental Load for Mutable Data)

Refresh directory listings after files were added/removed behind Alluxio's back

--with-index-service / --only-index-service (see Rebuilding the Listing Index)

Change a parameter and re-run after submitting

add --overwrite (see Changing a Submitted Job's Parameters)

Very large dataset (hundreds of thousands of files or more)

split into multiple jobs (see Splitting Very Large Datasets)

The examples below use --path; the two index-file forms are covered under Loading from an Index File.

Before You Submit

Three quick checks prevent most "everything failed" incidents:

  1. Mount-point path spelling — the target must be under a mounted UFS. A misspelled bucket makes every file fail instantly and the job FAIL within seconds with zero bytes loaded. Verify with alluxio mount list.

  2. A clean file list — entries that do not exist are each charged as one failed file (retried up to the cap, then given up); the rest still load, but the job ends FAILED. Clean the list beforehand when you can.

  3. Re-submits are idempotent — re-submitting the same target does not create a new job, whether the existing one is still running or was stopped; it merges into (or resumes) that job, which keeps its original parameters. To change parameters, use --overwrite (see Changing a Submitted Job's Parameters).

Submit and Monitor

--path accepts either a UFS path (e.g. s3://my-bucket/dataset/) or an Alluxio virtual path (e.g. /mnt/dataset/). See the CLI reference for details.

Example progress output:

Submitting via the REST API

Every job load operation is also available on the coordinator's REST endpoint (POST /api/v1/load on port 19999), which is convenient for programmatic submission. The request body fields mirror the CLI flags:

  • path — a single directory or file (like --path)

  • index — a UFS index file (like --index-file)

  • paths + alias — an inline path list (like --local-index-file); alias is a caller-chosen name that identifies the job for later progress/stop queries

  • isOverWrite — top-level, like --overwrite

  • options — a nested object: skipIfExists, loadPolicy ("IF_CHANGED"), verify, batchSize, fileFilterRegex, indexServiceMode ("withLoad" / "indexOnly", see Rebuilding the Listing Index)

For the full request/response schema, error codes, and the stop/file-list endpoints, see the REST API reference.

Stop a Running Job

A stopped job can be resumed by submitting it again with --submit. Already-cached files will be skipped if --skip-if-exists is included.

Key Flags

Flag
Description

--submit

Submit the job asynchronously (returns immediately)

--progress

Show progress of a submitted job

--stop

Stop a running job

--verify

After load completes, re-check every file and reload any that are not fully cached.

--replicas <n>

Load n replicas per file (default: 1); useful for high-concurrency reads

--skip-if-exists

Skip files that are already fully cached (safe to re-run a load job)

--load-policy IF_CHANGED

Re-check each cached file against UFS metadata; reload only files whose content has changed. Use for incremental sync of mutable datasets.

--metadata-only

Refresh file metadata without loading data; drops cached data of changed files. See Cache Invalidation

--batch-size <n>

Number of files per batch per worker. Default: 600. The default works well across file sizes; you normally do not need to change it.

--partial-listing

Start loading before the full directory listing completes; useful for very large directories

--index-file <ufs-path>

Load a specific list of files defined in a UFS index file (one path per line)

--local-index-file <local-path>

Like --index-file, but the manifest is read from the client's local filesystem and sent with the request — no UFS upload needed

--overwrite

Terminate an existing job for the same target and submit a fresh one with new parameters (see Changing a Submitted Job's Parameters)

--with-index-service

Rebuild the listing index for every directory in the loaded tree, concurrently with the data load (see Rebuilding the Listing Index)

--only-index-service

Only rebuild the listing index — no data or metadata is loaded (see Rebuilding the Listing Index)

For the full flag reference, see job load CLI documentation.

Loading from an Index File

For selective loading or when the directory tree is too large to traverse upfront:

Index file format — one UFS path per line, lines starting with # are comments:

Each line is one of two kinds:

  • A file (no trailing /) is dispatched to a worker directly — the coordinator does no listing for it, which is the lowest-overhead form.

  • A directory (trailing /) is enumerated recursively by the coordinator. If a listed entry does not exist, that single line is counted as one failed file and the remaining lines continue (see the Failure Modes section).

To submit a manifest that lives on the client's local filesystem (e.g., one generated by a script), use --local-index-file <local-path> instead — the client reads the file and sends it with the request, so it does not need to be uploaded to UFS first. The file format is identical. For very long manifests (hundreds of thousands of lines), prefer a UFS --index-file: a local manifest is sent in a single request, and a very large one can exceed the request-size limit.

Incremental Load for Mutable Data

When the underlying dataset changes periodically (e.g., daily model checkpoints, updated training splits), use --load-policy IF_CHANGED to sync only the files that have changed since the last load:

--load-policy IF_CHANGED re-checks UFS metadata for each already-cached file and reloads it only if the content has changed. Files that are not yet cached are loaded unconditionally. This makes it the right choice for periodic sync of a mutable dataset: new files get cached, changed files get refreshed, and unchanged files are skipped.

Flag
Cached files
Uncached files

--submit (no flags)

Reload unconditionally

Load

--skip-if-exists

Skip

Load

--load-policy IF_CHANGED

Reload only if changed

Load

Rebuilding the Listing Index (Index Service)

Available since 3.8-15.1.18.

The index service is Alluxio's distributed cache for directory listings: once a directory's listing is indexed, ls (CLI and FUSE) is served from the index instead of listing the UFS on every call, which is what keeps listings of very large directories fast. See Metadata Optimization for how it works and how to enable it.

When files are added or removed in the UFS behind Alluxio's back (e.g., another pipeline writes directly to S3), a cached directory listing does not notice: ls on that directory keeps returning the stale list until the listing cache is freed or expires. Two job load modes rebuild the listing index as part of the job, so the listings served to ls (CLI and FUSE) match the UFS again:

  • --with-index-service — load data/metadata as usual and rebuild the listing index of every directory in the loaded tree. The index rebuild runs concurrently with the data load, so for typical loads it adds little or no wall time.

  • --only-index-service — rebuild the listing index only; no data or metadata is loaded (Bytes Loaded: 0B). This is the lightweight way to refresh listings after an out-of-band change when you do not need the new data cached yet.

The two flags are mutually exclusive.

Progress reporting gains an extra line when either mode is active:

A directory that fails to index does not fail the job and is not counted in the file failure rate — the job reports SUCCEEDED with a note listing how many directories failed; re-run to retry them. Index rebuild failures are also exposed as the coordinator metric alluxio_distributed_load_index_dirs_total{status="failed"}.

Consistency after out-of-band UFS changes — what --with-index-service guarantees for a directory it rebuilds:

Out-of-band change
Result after the load

File added

Listing, metadata, and data are all consistent with UFS (any flag combination)

File overwritten

Consistent by default. With --skip-if-exists, the listing is refreshed but cached data/metadata keep the old version — add --load-policy IF_CHANGED to refresh all three.

File deleted

The file disappears from its parent's listing. Its already-cached metadata/data are not proactively removed — direct getStatus/reads of the old path may still hit cache until TTL/eviction or a job free.

To drop a previously built listing index (e.g., before measuring a cold load), use job free --path <path> --index-service --submit.

Two coordinator properties bound the index work per job:

  • alluxio.coordinator.dora.load.job.index.list.concurrency (default 4) — how many directories one job indexes concurrently. Each in-flight directory performs one single-level UFS listing; raise it to speed up indexing when the UFS has headroom.

  • alluxio.coordinator.dora.load.job.index.max.dirs (default 10000000) — cap on directories collected per job (~160 bytes of coordinator memory per directory); directories beyond the cap are counted as failed to index.

The rebuild cost scales with the number of entries listed, not bytes loaded: hidden behind any reasonably long data load it is effectively free, while an index-only job on a single directory with millions of entries takes minutes (bounded by UFS listing speed).

Changing a Submitted Job's Parameters

Re-submitting the same target is idempotent: a running job keeps running and a stopped one resumes, and either way the job keeps the parameters it was submitted with — only the batch size is picked up from the new submission. The CLI says so explicitly (Load already running ... Running job will continue running. Stopped job will resume to run, with batch size param updated. Other params will remain the same.), and the REST API answers 409 carrying the original job id. Stopping a job first therefore does not let you change anything: a stopped job is kept rather than discarded, so the next submission for that target resumes it.

A job's identity is its target path plus its type (load, free, ...), and the target is whichever of --path, --index-file or --local-index-file was submitted — the identity does not record which flag it came from. Submitting the same string through a different flag therefore lands on the same job: once --index-file s3://bucket/data has been submitted, --path s3://bucket/data resumes that index-file job instead of starting a directory load.

To change a submit-time parameter, add --overwrite: it terminates the existing job and submits a fresh one with the new parameters in one step. The example below raises --batch-size — one of the few cases where overriding the default helps, e.g. a small-file workload where workers sit mostly idle.

The old job is terminated and marked FAILED (reason: will be overwritten) — this is expected. --overwrite does not evict already-cached data (it is not job free), so combine it with --skip-if-exists: the new job re-enumerates every file, but the worker skips those already fully cached and reloads only the remainder.

Splitting Very Large Datasets

A single job over hundreds of thousands of files works, but splitting into several medium-sized jobs is usually a better experience: a smaller failure blast radius (one job's failure affects only its slice) and faster gap-filling re-runs. Split by directory (one job per partition), or split a manifest into groups (one --local-index-file per group). This is about failure isolation and gap-filling granularity, not concurrency — the coordinator caps the number of simultaneously running jobs, so a handful to a dozen parallel jobs is the right scale.

Integrating with ML Training

A typical workflow: load data → verify → run training.

Near-100% cache coverage: For critical datasets, run a second pass with --skip-if-exists after the first job reaches SUCCEEDED. In rare cases — transient worker failures or hash ring boundary timing — a single pass may miss a small fraction of files. A second pass fills those gaps without re-loading already-cached data:

Failure Modes

job load is a best-effort, long-running task; it does not guarantee 100% success and is not all-or-nothing. On large datasets, occasional per-file failures are normal (transient object-store timeouts, a worker restarting). A final Job State: FAILED means partially failed — successfully loaded files remain cached; it does not mean nothing was loaded. Each failed file is retried a bounded number of times (default 40, alluxio.coordinator.dora.load.job.subtask.max.retry.attempts) before being counted as failed, so failures do not retry forever or wedge the job. The standard recovery is to re-run with --skip-if-exists.

Job State: FAILED with Files Failed > 0

Check the file-level failure list:

Common causes: UFS access errors, network timeouts, or missing credentials. Fix the underlying issue, then resubmit with --skip-if-exists to avoid re-loading already-cached files (forgetting --skip-if-exists makes the re-run reload everything, which can take a long time). The Failed files saved to: <path> line at the top of the progress report points to a complete failed-file list written on the coordinator — use it to build a gap-filling index file containing only the failed entries; --progress --format JSON --verbose gives structured detail and recent failure-reason samples.

Job State: FAILED immediately after submit

Run --progress --verbose for detail:

Often caused by: path not found in mount table (verify with alluxio mount list), or insufficient cache quota.

Load succeeds but reads still go to UFS

Verify that specific files are actually cached:

If files show as uncached after a successful load, data may have been evicted. Check cache capacity and eviction settings — see Cache Eviction. For cluster-wide cache hit rate, see Monitoring.

check-cached operates on a directory or a manifest (pass --index-file <manifest> to check a specific list); it does not accept a single file path. To verify one individual file, query a worker directly and check that mInAlluxioPercentage is 100:

If the path contains = — e.g., Hive-style partition directories like date=2026-07-01 — encode it as %253D; otherwise the path is truncated and the query returns 404.

Troubleshooting quick reference:

Symptom
What to do

Ends FAILED, but most files read fine and fast

Normal partial failure; check Files Failed and the failure list, re-run with --skip-if-exists to fill the gaps

FAILED within seconds of submit, zero bytes loaded

Target is likely not under a mount point (misspelled bucket); verify the mounts and path

The list contains files that do not exist

Each is charged as one failure; the rest load normally; final state FAILED — cross-check with the failure list

A re-run takes a long time

You forgot --skip-if-exists (reloads everything); or gap-fill precisely from the failure list

Source files updated but reads still return old content

Refresh the path with --metadata-only, see Cache Invalidation

Confirm whether a specific file is cached

Single file: worker /v1/info; batch: check-cached --index-file (see above)

Retention of Historical Jobs

Completed job records are kept for a configurable period. The default is 7 days. To adjust:

Related

Last updated