> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/cache/cache-invalidation.md). # Cache Invalidation You changed something in your storage — deleted a file, overwrote it, replaced it with a different version — but Alluxio keeps serving the old copy, without any error to tell you so. This is expected behavior, not a bug. This page explains why it happens and how to bring the cache back in line with your storage. If instead you are trying to reclaim cache space, see [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md). ## Do I Have This Problem? Match what you are seeing, then follow the link. | What you observe | Where to go | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Reads on the same paths keep failing or returning wrong data for hours, and **restarting Alluxio did not help** | [Why Cached Data Can Differ From Storage](#why-cached-data-can-differ-from-storage). A restart does not clear it: the cached copy is on disk and survives one | | You overwrite files in place, for example in a daily job, and want Alluxio to pick up the new content | [Metadata-Only Load After an Overwrite](#metadata-only-load-after-an-overwrite) | | A file comes back shorter than its own `Content-Length`, or with old and new content mixed together | [How a Cut-Off Read Looks](#how-a-cut-off-read-looks) | | A FUSE read fails with `EIO`, or a GET with `If-Match` returns `412` | [Troubleshooting](#troubleshooting) | | `ls` shows files you deleted, or is missing files you added | [Directory Listings](#directory-listings) | | Reads are slow, but the content is correct | Not staleness. See [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md) | Before clearing anything, confirm it. Ask Alluxio what it is serving: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio fs stat s3://bucket/dataset/model.bin ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio fs stat s3://bucket/dataset/model.bin ``` {% endtab %} {% endtabs %} then ask your storage what it actually holds: ```shell aws s3api head-object --bucket bucket --key dataset/model.bin --query '[ContentLength,ETag]' ``` Compare two fields. If `length` or `contentHash` differs from the object's size and ETag, Alluxio is holding an older version. **`contentHash` is the one that settles it.** A file overwritten by a same-sized replacement has an identical `length`, so only the hash shows the difference. `source=READ_CACHE_WORKER` in the same output confirms the answer came from cache rather than a fresh look at your storage. ## Why Cached Data Can Differ From Storage Alluxio caches three things, each on its own: * **File data**, the pages of the file. * **File metadata**: the length, the modification time and the content hash. Together they are the file's **version**. * **Directory listings**, the names in a directory. **Alluxio never watches your storage for changes.** There is no notification from S3 or HDFS telling Alluxio that a file changed, and Alluxio does not poll to find out. By default it assumes cached data never changes (the `immutable` cache policy) and does not look again. That is right for model weights and versioned datasets, and it is the fastest option. Writing through Alluxio is fine too, because Alluxio updates its own cache on those writes. The cache goes stale only when data is changed **directly in your storage**, bypassing Alluxio. Then: | What you did in your storage | What reads return | | ---------------------------- | -------------------------------- | | Overwrote a file | The old content | | Changed its size | The old content, at the old size | | Deleted a file | The old content, still found | A file that is only partly cached can come back as a mix of old and new content.
Two tracks over time. The Alluxio cache and your storage both hold version 1 of a file. You overwrite the file, and your storage now holds version 2, but the cache still holds version 1, so reads return the old content. After a metadata-only load, job free or a maxAge expiry on a read, the cache holds version 2 as well.

After you overwrite a file in your storage, the cache keeps the old version until you run a metadata-only load or job free, or a maxAge policy expires on a read.

{% hint style="warning" %} **These failures are silent.** A cached read of a file you deleted still returns `HTTP 200` and the old bytes — there is no error to alert you. It gets more confusing when tools disagree: after you replace a file with a larger one, `ls` reports the **new** size (it asks your storage) while a read still returns the **old, shorter** content from cache. If an application is reading data you changed underneath it, assume it is reading the old version until you invalidate. {% endhint %} Directory listings go stale in the same way, but are fixed with a different command. See [Directory Listings](#directory-listings). ## Ways to Bring the Cache Back in Line There are two kinds. **Passive** ways make Alluxio re-check on its own when a file is read. **Active** ways are commands you run after you change your storage. | | Way | What it does | Limit | | ------- | -------------------------------------- | ----------------------------------------------------- | --------------------------------------- | | Passive | `maxAge` cache policy | Re-checks storage on the first read after N minutes | Stale for up to N minutes | | Passive | `skipCache` cache policy | Reads from storage every time | No caching benefit | | Passive | `alluxio.snapshot.consistency.enabled` | Checks every read against the file's version | Guards reads; does not update the cache | | Active | `job free` | Drops the cached copy; the next read fetches it again | Re-reads everything it dropped | | Active | `job load --metadata-only` | Drops the cached data of files that changed | Run it after each overwrite | | Active | `job load` | Reloads the data into the cache | Re-reads every file | Pick by how your data behaves: | How your data behaves | Use | | -------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | Never changes once written (model weights, versioned datasets) | Nothing. The default `immutable` policy is the fastest | | Updated on a schedule you know (daily ETL) | `maxAge` set to that interval | | Overwritten in place by a job you control | A [metadata-only load](#metadata-only-load-after-an-overwrite) after each overwrite | | Changes unpredictably and must never be stale | `skipCache`, or a short `maxAge` | | Files were deleted | `job free` | | Readers must never see old and new content mixed | Also turn on [snapshot consistency](#snapshot-consistency) | ## Active: You Tell Alluxio These commands do nothing until you run them. Use them after you change your storage. ### Free the Cached Copy Run `job free` on the affected path. It clears Alluxio's cached copy — both the file content and what Alluxio remembers about the file (whether it exists, how big it is) — so the next read goes back to your storage and picks up the current truth. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # One file kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job free --path s3://bucket/dataset/model.bin --submit # A whole directory kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job free --path s3://bucket/dataset/ --submit --recursive # Check that it finished kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job free --path s3://bucket/dataset/ --progress ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # One file bin/alluxio job free --path s3://bucket/dataset/model.bin --submit # A whole directory bin/alluxio job free --path s3://bucket/dataset/ --submit --recursive # Check that it finished bin/alluxio job free --path s3://bucket/dataset/ --progress ``` {% endtab %} {% endtabs %} Your data in the storage system is not touched — only Alluxio's cached copy is dropped. After the job succeeds: * A file you **deleted** now correctly reports "not found" * A file you **changed** is re-read at its current content and size ### Metadata-Only Load After an Overwrite [`job free`](#free-the-cached-copy) drops the cached copy of every file under the path, changed or not. If you overwrote some files in place and the rest should stay cached, run a metadata-only load instead. It re-reads each file's metadata from your storage and drops the cached data of every file whose length or content hash changed. Each run lists the target path, so point it at the directories that changed, not at the whole bucket. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # Metadata-only load kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path s3://bucket/dataset/ --metadata-only --submit # Check that it finished kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio job load --path s3://bucket/dataset/ --progress ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # Metadata-only load bin/alluxio job load --path s3://bucket/dataset/ --metadata-only --submit # Check that it finished bin/alluxio job load --path s3://bucket/dataset/ --progress ``` {% endtab %} {% endtabs %} Run it after each overwrite, whether or not `alluxio.snapshot.consistency.enabled` is set. That setting only guards reads that reach your storage and does not replace the metadata-only load (see [Snapshot Consistency](#snapshot-consistency)). Keep in mind: * Reads of a changed file return the old content until the job finishes. * A load job is identified by its path. If one is running or stopped on the same path, submitting again resumes it and the metadata-only load does not run; see [Changing a Submitted Job's Parameters](/ee-ai-en/ai-3.8-15.1.x/cache/cache-loading.md#changing-a-submitted-jobs-parameters). * Do not add `--skip-if-exists`: fully cached files are skipped, so they are never checked. * Deleted files are not covered; use `job free` for those. * With `alluxio.user.file.segment.enabled=true` it needs AI-3.8-15.1.26 or later. Earlier releases update only the worker that holds the first segment. ### Reload the Data `job load` reads the files from your storage again and caches them, which also warms the cache. It reloads every file, changed or not. For how to run it, see [Cache Loading](/ee-ai-en/ai-3.8-15.1.x/cache/cache-loading.md). ## Passive: Alluxio Notices on a Read These need no command. Alluxio handles them when a file is read. ### maxAge and skipCache A **cache filter policy** is a rule you attach to a path. It decides whether Alluxio may keep using its cached copy without re-checking your storage. The default, `immutable`, never re-checks. `maxAge` re-checks on the first read after the age you set, and `skipCache` reads from your storage every time. How to write and apply these rules, and how they differ from TTL, is in [Cache Policies](/ee-ai-en/ai-3.8-15.1.x/cache/cache-policies.md#controlling-what-to-cache-cache-filter-policies). To see which rule is in effect on a path: {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio fs ls -c ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell bin/alluxio fs ls -c ``` {% endtab %} {% endtabs %} ### Snapshot Consistency ```properties alluxio.snapshot.consistency.enabled=true ``` With it on, every read checks the bytes it returns against the version of the file it started with. A read that would return old and new content mixed is cut off instead, and the worker drops the stale copy so the next read is correct. It guards reads. It does not update the cache: a file that is fully cached and was overwritten is not detected, because the read never contacts your storage. Use it together with a metadata-only load or a `maxAge` policy. See [Snapshot Consistency in Detail](#snapshot-consistency-in-detail) for what a read returns on S3 and FUSE, the requirements and the cost. ## Directory Listings Alluxio caches file data and directory listings separately, and each has its own command. Clearing one does nothing for the other. If `ls` shows files you deleted, or is missing files you added, the listing is stale, while the files themselves may still read fine by their full path. {% tabs %} {% tab title="Kubernetes (Operator)" %} ```shell # One directory kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio index invalidate --path s3://bucket/tables/sales # Everything underneath — slow on a large tree, prefer targeting directories directly kubectl exec -n alluxio-cluster-coordinator-0 -- \ alluxio index invalidate -R --path s3://bucket/tables/ ``` {% endtab %} {% tab title="Docker / Bare-Metal" %} ```shell # One directory bin/alluxio index invalidate --path s3://bucket/tables/sales # Everything underneath — slow on a large tree, prefer targeting directories directly bin/alluxio index invalidate -R --path s3://bucket/tables/ ``` {% endtab %} {% endtabs %} The next `ls` of that directory loads fresh from your storage. See [Metadata Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/metadata-listing.md) if listings are not cached on your cluster yet. If one application still sees stale results after you invalidated on the cluster, check whether it has its own client-side metadata cache enabled (`alluxio.user.metadata.cache.*`, off by default) — neither command reaches that one. ## Snapshot Consistency in Detail This is the reference for the setting summarized under [Snapshot Consistency](#snapshot-consistency). It covers the requirements, what a read returns, and how to troubleshoot it. ### Requirements {% hint style="info" %} Requires AI-3.8-15.1.18 or later, and AI-3.8-15.1.27 or later on GCS: earlier releases abort every GCS read when the setting is on, including reads of objects that did not change. Set it on the workers and the clients, and restart them to pick it up. {% endhint %} ### What a Read Returns After an in-place overwrite, the setting changes what a read returns until you run the metadata-only load. The fix is the same either way: the [metadata-only load](#metadata-only-load-after-an-overwrite). | After the overwrite | Setting off (default) | Setting on | | -------------------------------------- | ------------------------- | --------------------------------- | | Read of a fully cached file | Old content, no error | Old content, no error | | Read that needs data from your storage | New and old bytes spliced | Cut off; the next read is correct | | After the metadata-only load | New content | New content | | What it costs | Nothing | One cut-off read per changed file | #### How a Cut-Off Read Looks Each read records the version of the object it is serving — on S3, the ETag — and checks every byte it returns against that version. If the object was replaced, the read is **cut off** rather than completed with a mix of old and new content. How that surfaces matters, because it is not an HTTP error. The response has already started, so it keeps its `200` status and its declared `Content-Length` — 12582912 for the 12 MiB object below — and the body simply stops early. The same read, of an object overwritten while one of its pages was cached: | | Setting off | Setting on | | ---------------- | -------------------------------------- | ------------------------------------------- | | Bytes delivered | 12582912 | 3145728 | | Content | first page old, rest new — **spliced** | old bytes only, then nothing | | `curl` exit code | `0` | `18` (transfer closed with bytes remaining) | Where it stops is not itself meaningful — it is however much had already reached the network, not a page or object boundary. What matters is that it stops short of the declared length. So the setting does not remove the truncation — the truncation is *how* it avoids serving a splice. **Any client that checks `Content-Length` against what it received sees the failure**: `curl` exits non-zero, an HTTP library raises an incomplete-read error, an S3 SDK raises a retryable transfer error. A client that ignores transfer errors and keeps whatever it buffered ends up with a short file it believes is complete. The interruption also repairs the cache: the worker drops the stale copy, so the next read returns the current object. Over ten consecutive reads of an overwritten file: one cut-off transfer, then nine correct ones, and never a spliced response. One thing to know before enabling it: **one read is cut off per changed file.** That interruption *is* the detection, so it is expected rather than a fault. That puts a requirement on your applications: one that retries on transfer errors sees a transient failure followed by correct data; one that treats any I/O error as fatal turns a self-healing event into a job failure. ### Which Version a Read Serves With the setting on, a read fixes the version it serves when it starts, and every byte it returns matches that version (the table above shows what the setting off does instead): * **S3 API:** each GetObject fixes its version once, from the file's metadata on the worker that serves it. * **FUSE and other file clients:** the version is fixed when the file is opened and applies to every read on that open file until it is closed. If the file is overwritten and a metadata-only load runs after the open, a read that needs data from your storage fails (`EIO` on FUSE) instead of returning the new version. Reopen the file to read it. Nothing ties separate requests, or separate opens, to the same version, so a client that downloads one object in several ranged GETs can mix versions if the object is overwritten and a metadata-only load runs in between. Over the S3 API, send `If-Match` with the ETag from the first response on every later request. Once Alluxio's metadata reflects the change, they fail with `412 Precondition Failed` instead of returning another version. This needs AI-3.8-15.1.20 or later; earlier releases do not evaluate the header. The check does not depend on the setting. ### Troubleshooting #### Reads Stop Short of Content-Length **Likely cause:** the object was overwritten while some of its pages were cached. The read was cut off on purpose, so that it does not return old and new content mixed. **What to do:** retry the read. The worker dropped the stale copy, so the next read returns the current object. #### A FUSE Read Fails With EIO **Likely cause:** the file was overwritten and a metadata-only load ran after you opened it. A read on that open file needs data from your storage, and the data no longer matches the version the file was opened with. **What to do:** reopen the file. #### A GET With If-Match Returns 412 **Likely cause:** the object changed since the response whose ETag you sent. This is what `If-Match` is for. It needs AI-3.8-15.1.20 or later. **What to do:** start the download again from the beginning. #### Every GCS Read Fails **Likely cause:** the release is older than AI-3.8-15.1.27. With the setting on, those releases abort every GCS read, including reads of objects that did not change. **What to do:** upgrade, or turn the setting off. ## Related Pages * [Cache Policies](/ee-ai-en/ai-3.8-15.1.x/cache/cache-policies.md): how to write and apply cache filter rules, TTL, pinning, quotas * [Cache Loading](/ee-ai-en/ai-3.8-15.1.x/cache/cache-loading.md): the full `job load` reference * [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md): freeing cache space, and the complete `job free` reference * [Metadata Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/metadata-listing.md): directory-listing cache setup and tuning * [S3 API](/ee-ai-en/ai-3.8-15.1.x/data-access/s3-api.md): conditional requests such as `If-Match`