Ctrlk
ProductsLanguageHome
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Cache Invalidation

You changed something in your storage — deleted a file, overwrote it, replaced it with a different version — but Alluxio keeps serving the old copy, without any error to tell you so.

This is expected behavior, not a bug. This page explains why it happens and how to bring the cache back in line with your storage.

If instead you are trying to reclaim cache space, see Cache Eviction.

Do I Have This Problem?

Match what you are seeing, then follow the link.

What you observe
Where to go

Reads on the same paths keep failing or returning wrong data for hours, and restarting Alluxio did not help

Why Cached Data Can Differ From Storage. A restart does not clear it: the cached copy is on disk and survives one

You overwrite files in place, for example in a daily job, and want Alluxio to pick up the new content

Metadata-Only Load After an Overwrite

A file comes back shorter than its own Content-Length, or with old and new content mixed together

How a Cut-Off Read Looks

A FUSE read fails with EIO, or a GET with If-Match returns 412

Troubleshooting

ls shows files you deleted, or is missing files you added

Directory Listings

Reads are slow, but the content is correct

Not staleness. See Cache Eviction

Before clearing anything, confirm it. Ask Alluxio what it is serving:

kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio fs stat s3://bucket/dataset/model.bin
bin/alluxio fs stat s3://bucket/dataset/model.bin

then ask your storage what it actually holds:

aws s3api head-object --bucket bucket --key dataset/model.bin --query '[ContentLength,ETag]'

Compare two fields. If length or contentHash differs from the object's size and ETag, Alluxio is holding an older version.

contentHash is the one that settles it. A file overwritten by a same-sized replacement has an identical length, so only the hash shows the difference.

source=READ_CACHE_WORKER in the same output confirms the answer came from cache rather than a fresh look at your storage.

Why Cached Data Can Differ From Storage

Alluxio caches three things, each on its own:

  • File data, the pages of the file.

  • File metadata: the length, the modification time and the content hash. Together they are the file's version.

  • Directory listings, the names in a directory.

Alluxio never watches your storage for changes. There is no notification from S3 or HDFS telling Alluxio that a file changed, and Alluxio does not poll to find out. By default it assumes cached data never changes (the immutable cache policy) and does not look again.

That is right for model weights and versioned datasets, and it is the fastest option. Writing through Alluxio is fine too, because Alluxio updates its own cache on those writes. The cache goes stale only when data is changed directly in your storage, bypassing Alluxio. Then:

What you did in your storage
What reads return

Overwrote a file

The old content

Changed its size

The old content, at the old size

Deleted a file

The old content, still found

A file that is only partly cached can come back as a mix of old and new content.

Two tracks over time. The Alluxio cache and your storage both hold version 1 of a file. You overwrite the file, and your storage now holds version 2, but the cache still holds version 1, so reads return the old content. After a metadata-only load, job free or a maxAge expiry on a read, the cache holds version 2 as well.
After you overwrite a file in your storage, the cache keeps the old version until you run a metadata-only load or job free, or a maxAge policy expires on a read.

These failures are silent. A cached read of a file you deleted still returns HTTP 200 and the old bytes — there is no error to alert you. It gets more confusing when tools disagree: after you replace a file with a larger one, ls reports the new size (it asks your storage) while a read still returns the old, shorter content from cache. If an application is reading data you changed underneath it, assume it is reading the old version until you invalidate.

Directory listings go stale in the same way, but are fixed with a different command. See Directory Listings.

Ways to Bring the Cache Back in Line

There are two kinds. Passive ways make Alluxio re-check on its own when a file is read. Active ways are commands you run after you change your storage.

Way
What it does
Limit

Passive

maxAge cache policy

Re-checks storage on the first read after N minutes

Stale for up to N minutes

Passive

skipCache cache policy

Reads from storage every time

No caching benefit

Passive

alluxio.snapshot.consistency.enabled

Checks every read against the file's version

Guards reads; does not update the cache

Active

job free

Drops the cached copy; the next read fetches it again

Re-reads everything it dropped

Active

job load --metadata-only

Drops the cached data of files that changed

Run it after each overwrite

Active

job load

Reloads the data into the cache

Re-reads every file

Pick by how your data behaves:

How your data behaves
Use

Never changes once written (model weights, versioned datasets)

Nothing. The default immutable policy is the fastest

Updated on a schedule you know (daily ETL)

maxAge set to that interval

Overwritten in place by a job you control

A metadata-only load after each overwrite

Changes unpredictably and must never be stale

skipCache, or a short maxAge

Files were deleted

job free

Readers must never see old and new content mixed

Also turn on snapshot consistency

Active: You Tell Alluxio

These commands do nothing until you run them. Use them after you change your storage.

Free the Cached Copy

Run job free on the affected path. It clears Alluxio's cached copy — both the file content and what Alluxio remembers about the file (whether it exists, how big it is) — so the next read goes back to your storage and picks up the current truth.

Your data in the storage system is not touched — only Alluxio's cached copy is dropped.

After the job succeeds:

  • A file you deleted now correctly reports "not found"

  • A file you changed is re-read at its current content and size

Metadata-Only Load After an Overwrite

job free drops the cached copy of every file under the path, changed or not. If you overwrote some files in place and the rest should stay cached, run a metadata-only load instead. It re-reads each file's metadata from your storage and drops the cached data of every file whose length or content hash changed. Each run lists the target path, so point it at the directories that changed, not at the whole bucket.

Run it after each overwrite, whether or not alluxio.snapshot.consistency.enabled is set. That setting only guards reads that reach your storage and does not replace the metadata-only load (see Snapshot Consistency).

Keep in mind:

  • Reads of a changed file return the old content until the job finishes.

  • A load job is identified by its path. If one is running or stopped on the same path, submitting again resumes it and the metadata-only load does not run; see Changing a Running Job's Parameters.

  • Do not add --skip-if-exists: fully cached files are skipped, so they are never checked.

  • Deleted files are not covered; use job free for those.

  • With alluxio.user.file.segment.enabled=true it needs AI-3.8-15.1.26 or later. Earlier releases update only the worker that holds the first segment.

Reload the Data

job load reads the files from your storage again and caches them, which also warms the cache. It reloads every file, changed or not. For how to run it, see Cache Loading.

Passive: Alluxio Notices on a Read

These need no command. Alluxio handles them when a file is read.

maxAge and skipCache

A cache filter policy is a rule you attach to a path. It decides whether Alluxio may keep using its cached copy without re-checking your storage. The default, immutable, never re-checks. maxAge re-checks on the first read after the age you set, and skipCache reads from your storage every time. How to write and apply these rules, and how they differ from TTL, is in Cache Policies.

To see which rule is in effect on a path:

Snapshot Consistency

With it on, every read checks the bytes it returns against the version of the file it started with. A read that would return old and new content mixed is cut off instead, and the worker drops the stale copy so the next read is correct.

It guards reads. It does not update the cache: a file that is fully cached and was overwritten is not detected, because the read never contacts your storage. Use it together with a metadata-only load or a maxAge policy.

See Snapshot Consistency in Detail for what a read returns on S3 and FUSE, the requirements and the cost.

Directory Listings

Alluxio caches file data and directory listings separately, and each has its own command. Clearing one does nothing for the other. If ls shows files you deleted, or is missing files you added, the listing is stale, while the files themselves may still read fine by their full path.

The next ls of that directory loads fresh from your storage. See Metadata Optimization if listings are not cached on your cluster yet.

If one application still sees stale results after you invalidated on the cluster, check whether it has its own client-side metadata cache enabled (alluxio.user.metadata.cache.*, off by default) — neither command reaches that one.

Snapshot Consistency in Detail

This is the reference for the setting summarized under Snapshot Consistency. It covers the requirements, what a read returns, and how to troubleshoot it.

Requirements

Requires AI-3.8-15.1.18 or later, and AI-3.8-15.1.27 or later on GCS: earlier releases abort every GCS read when the setting is on, including reads of objects that did not change. Set it on the workers and the clients, and restart them to pick it up.

What a Read Returns

After an in-place overwrite, the setting changes what a read returns until you run the metadata-only load. The fix is the same either way: the metadata-only load.

After the overwrite
Setting off (default)
Setting on

Read of a fully cached file

Old content, no error

Old content, no error

Read that needs data from your storage

New and old bytes spliced

Cut off; the next read is correct

After the metadata-only load

New content

New content

What it costs

Nothing

One cut-off read per changed file

How a Cut-Off Read Looks

Each read records the version of the object it is serving — on S3, the ETag — and checks every byte it returns against that version. If the object was replaced, the read is cut off rather than completed with a mix of old and new content.

How that surfaces matters, because it is not an HTTP error. The response has already started, so it keeps its 200 status and its declared Content-Length — 12582912 for the 12 MiB object below — and the body simply stops early.

The same read, of an object overwritten while one of its pages was cached:

Setting off
Setting on

Bytes delivered

12582912

3145728

Content

first page old, rest new — spliced

old bytes only, then nothing

curl exit code

0

18 (transfer closed with bytes remaining)

Where it stops is not itself meaningful — it is however much had already reached the network, not a page or object boundary. What matters is that it stops short of the declared length.

So the setting does not remove the truncation — the truncation is how it avoids serving a splice.

Any client that checks Content-Length against what it received sees the failure: curl exits non-zero, an HTTP library raises an incomplete-read error, an S3 SDK raises a retryable transfer error.

A client that ignores transfer errors and keeps whatever it buffered ends up with a short file it believes is complete.

The interruption also repairs the cache: the worker drops the stale copy, so the next read returns the current object. Over ten consecutive reads of an overwritten file: one cut-off transfer, then nine correct ones, and never a spliced response.

One thing to know before enabling it: one read is cut off per changed file. That interruption is the detection, so it is expected rather than a fault.

That puts a requirement on your applications: one that retries on transfer errors sees a transient failure followed by correct data; one that treats any I/O error as fatal turns a self-healing event into a job failure.

Which Version a Read Serves

With the setting on, a read fixes the version it serves when it starts, and every byte it returns matches that version (the table above shows what the setting off does instead):

  • S3 API: each GetObject fixes its version once, from the file's metadata on the worker that serves it.

  • FUSE and other file clients: the version is fixed when the file is opened and applies to every read on that open file until it is closed. If the file is overwritten and a metadata-only load runs after the open, a read that needs data from your storage fails (EIO on FUSE) instead of returning the new version. Reopen the file to read it.

Nothing ties separate requests, or separate opens, to the same version, so a client that downloads one object in several ranged GETs can mix versions if the object is overwritten and a metadata-only load runs in between. Over the S3 API, send If-Match with the ETag from the first response on every later request. Once Alluxio's metadata reflects the change, they fail with 412 Precondition Failed instead of returning another version. This needs AI-3.8-15.1.20 or later; earlier releases do not evaluate the header. The check does not depend on the setting.

Troubleshooting

Reads Stop Short of Content-Length

Likely cause: the object was overwritten while some of its pages were cached. The read was cut off on purpose, so that it does not return old and new content mixed.

What to do: retry the read. The worker dropped the stale copy, so the next read returns the current object.

A FUSE Read Fails With EIO

Likely cause: the file was overwritten and a metadata-only load ran after you opened it. A read on that open file needs data from your storage, and the data no longer matches the version the file was opened with.

What to do: reopen the file.

A GET With If-Match Returns 412

Likely cause: the object changed since the response whose ETag you sent. This is what If-Match is for. It needs AI-3.8-15.1.20 or later.

What to do: start the download again from the beginning.

Every GCS Read Fails

Likely cause: the release is older than AI-3.8-15.1.27. With the setting on, those releases abort every GCS read, including reads of objects that did not change.

What to do: upgrade, or turn the setting off.

Related Pages

Last updated