Ctrlk
ProductsHome
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Cache Eviction

Data leaves the Alluxio cache in three ways:

  1. Automatic eviction — workers evict data when cache fills up, ordered by the configured policy (LRU by default)

  2. TTL expiry — background scan removes data whose lifetime has elapsed, regardless of access or priority

  3. Manual eviction — job free explicitly purges a path on demand

Automatic Eviction

When a worker needs space for new data, it runs an evictor to select which cached pages to remove. Three policies are available:

Policy
Evicts

LRU (default)

Data not accessed for the longest time

LFU

Data accessed the fewest times overall

FIFO

Data written earliest

To change the eviction policy, set in alluxio-site.properties on all workers:

# Use LFU instead of the default LRU
alluxio.worker.page.store.evictor.type=LFU

Asynchronous Eviction

By default, eviction runs synchronously during writes, which can add latency. Asynchronous eviction runs in the background to keep headroom available before the cache fills up:

alluxio.worker.page.store.async.eviction.enabled=true
# start evicting when cache usage exceeds this threshold (default: 0.9)
alluxio.worker.page.store.async.eviction.high.watermark=0.85
# stop evicting when cache usage drops below this threshold (default: 0.8)
alluxio.worker.page.store.async.eviction.low.watermark=0.75
# how often to check cache usage (default: 1min)
alluxio.worker.page.store.async.eviction.check.interval=30s

TTL-based eviction and Cache Priority also affect what gets evicted and when. See Cache Policies for details.

Manual Eviction: The free Job

Use job free to explicitly purge a path from the cache — without touching the underlying UFS data. For every file under the path, free removes both the cached data (pages) and the cached metadata (file status) from all workers, so the next access re-fetches everything from the UFS. Common scenarios:

  • Model version update: free the old version before (or after) loading the new one

  • Post-job cleanup: release space after a batch job completes

  • Force re-cache: free then reload to pick up UFS changes for an immutable-policy path

  • Out-of-band UFS changes: files deleted, overwritten, or changed directly in the UFS leave stale data and metadata behind — see Cache Invalidation for the full decision guide

If your goal is consistency with the UFS rather than reclaiming space — stale reads or errors after data changed outside Alluxio — start at Cache Invalidation. It covers when a policy will refresh the data on its own, and which of job free and index invalidate to reach for.

Submit and Monitor

Example progress output:

Freeing by Manifest (--ufs-index-file)

For a large or scattered set of paths, submit a manifest instead of a single --path: a plain text file on the UFS, one path per line. A line ending with / is treated as a directory and freed recursively on every worker; a line without a trailing / is a single file, routed to the worker that owns it. The trailing slash is significant — get it wrong and the entry is looked up as the wrong kind.

The coordinator streams the manifest in bounded rounds, so its size is not limited by memory; this is also the recommended mode for very large directories (see Rate Limiting below).

Rate Limiting

Free deletes metadata entries and cache pages on every worker, which lands on the worker's metadata store (RocksDB) and disk. To keep a large free job from interfering with serving traffic, the release rate is limited per worker by alluxio.job.free.rate.limit (default 20000 files per second per worker; 0 means unlimited). The limit is shared across all free jobs running on a worker, so the aggregate rate stays bounded. Override it per job with --rate-limit:

How to pick a value:

  • The default (20000/s per worker) sits near a worker's natural single-threaded delete rate, so most jobs need no tuning. Lower it to protect online workloads; 0 removes the cap.

  • The limit counts files, but the disk sees rate × pages-per-file — for datasets of large files (many pages each), scale the value down proportionally.

  • When online workloads share the cache disk, watch business read latency and worker disk IO, and reduce the rate if they degrade; in a maintenance window you can be aggressive.

  • --path mode on very large directories (tens of millions of files): each worker walks the path with a single thread, so the scan itself — not the rate limit — becomes the bottleneck. Switch to a manifest (--ufs-index-file, one subdirectory or file per line), which distributes entries across workers.

Note: --batch-size only controls how many index entries are dispatched to each worker per scheduling round, and only applies in manifest mode (--ufs-index-file/--paths); the release rate is governed by --rate-limit.

Stop a Running Free Job

Stopping leaves partially-freed data in the cache. The job can be resumed by submitting it again with --submit.

Version Update Pattern

To replace a pinned dataset with a newer version:

For a complete list of job free flags, see the job free CLI reference.

You can also trigger and manage free jobs via the REST API.

Stale Cache Cleaning

Cluster topology changes can leave data cached on workers that no longer "own" that data according to the consistent hash ring. This stale data consumes space but is never served to clients.

When this happens:

  • Workers are added or removed (ownership redistributes)

  • A file's replication factor is reduced

  • A worker goes offline temporarily and its data migrates, then it rejoins

Stale cache cleaning runs only on demand — there is no automatic or periodic scan. Two operations are exposed through the /cache REST endpoint:

  • scan-stale measures how much stale data each worker holds and logs a summary — it deletes nothing.

  • clear-stale deletes the stale data to reclaim space.

Run scan-stale first to gauge the impact, then clear-stale to reclaim the space.

Measure Stale Data (scan-only)

Each worker scans its local storage, compares ownership against the current hash ring, and logs a summary — nothing is deleted. Look in the worker logs for a line of the form Stale cache scan finished: scanned <N> files, found <N> stale bytes, removed <N> bytes (removed is 0 in scan-only mode).

Clear Stale Data

This submits an async job to each worker. Workers scan local storage, verify ownership against the current hash ring, and delete any data they no longer own. Monitor progress via the alluxio_cleared_stale_cached_data Prometheus metric or the worker-log summary above.

For more details, see the REST API reference.

clear-stale removes nothing while the write cache is enabled; run scan-stale to measure stale data instead.

Mass file deletion from stale cleaning can create metadata I/O pressure that delays Worker Pod termination. Run during a maintenance window on large clusters.

Last updated