> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/cache/cache-invalidation.md).

# Cache Invalidation

You changed something in your storage — deleted a file, overwrote it, replaced it with a different version — but Alluxio keeps serving the old copy, without any error to tell you so.

This is expected behavior, not a bug, and this page explains how to fix it.

If instead you are trying to reclaim cache space, see [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md).

## Do I Have This Problem?

Match what you are seeing — the first two rows are this page, the others are not.

| What you observe                                                                                                | Where to go                                                                           |
| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Reads on the same paths keep failing or returning wrong data for hours, and **restarting Alluxio did not help** | This page. A restart does not clear it: the cached copy is on disk and survives one.  |
| A file comes back shorter than its own `Content-Length`, or with old and new content mixed together             | This page — and it can be completely silent.                                          |
| `ls` shows files you deleted, or is missing files you added                                                     | The directory-listing cache — [`index invalidate`](#fixing-a-stale-directory-listing) |
| Reads are slow, but the content is correct                                                                      | Not staleness — see [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md) |

Before clearing anything, confirm it. Ask Alluxio what it is serving:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio fs stat s3://bucket/dataset/model.bin
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio fs stat s3://bucket/dataset/model.bin
```

{% endtab %}
{% endtabs %}

then ask your storage what it actually holds:

```shell
aws s3api head-object --bucket bucket --key dataset/model.bin --query '[ContentLength,ETag]'
```

Compare two fields. If `length` or `contentHash` differs from the object's size and ETag, Alluxio is holding an older version.

**`contentHash` is the one that settles it.** A file overwritten by a same-sized replacement has an identical `length`, so only the hash shows the difference.

`source=READ_CACHE_WORKER` in the same output confirms the answer came from cache rather than a fresh look at your storage.

## Which Cache Is Stale?

Alluxio caches **file data** and **directory listings** separately, and each has its own command. That is the whole reason there are two: clearing one does nothing for the other.

| Symptom                                                  | Stale cache                            | Fix                |
| -------------------------------------------------------- | -------------------------------------- | ------------------ |
| Reading a file returns old content, wrong size, or fails | the file's data, and its size and hash | `job free`         |
| `ls` shows deleted files, or is missing new ones         | the listing of names in a directory    | `index invalidate` |

`job free` handles a file's data and what Alluxio remembers about it together, which is why it is the answer for most cases.

The two really can go stale separately. A file that reads correctly but shows up in a listing it should not means only the listing is stale; a file missing from `ls` may still read fine by its full path.

## Fix It Now

Run `job free` on the affected path. It clears Alluxio's cached copy — both the file content and what Alluxio remembers about the file (whether it exists, how big it is) — so the next read goes back to your storage and picks up the current truth.

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# One file
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job free --path s3://bucket/dataset/model.bin --submit

# A whole directory
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job free --path s3://bucket/dataset/ --submit --recursive

# Check that it finished
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio job free --path s3://bucket/dataset/ --progress
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# One file
bin/alluxio job free --path s3://bucket/dataset/model.bin --submit

# A whole directory
bin/alluxio job free --path s3://bucket/dataset/ --submit --recursive

# Check that it finished
bin/alluxio job free --path s3://bucket/dataset/ --progress
```

{% endtab %}
{% endtabs %}

Your data in the storage system is not touched — only Alluxio's cached copy is dropped.

After the job succeeds:

* A file you **deleted** now correctly reports "not found"
* A file you **changed** is re-read at its current content and size

### Fixing a Stale Directory Listing

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
# One directory
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio index invalidate --path s3://bucket/tables/sales

# Everything underneath — slow on a large tree, prefer targeting directories directly
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio index invalidate -R --path s3://bucket/tables/
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
# One directory
bin/alluxio index invalidate --path s3://bucket/tables/sales

# Everything underneath — slow on a large tree, prefer targeting directories directly
bin/alluxio index invalidate -R --path s3://bucket/tables/
```

{% endtab %}
{% endtabs %}

The next `ls` of that directory loads fresh from your storage. See [Metadata Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/metadata-listing.md) if listings are not cached on your cluster yet.

If one application still sees stale results after you invalidated on the cluster, check whether it has its own client-side metadata cache enabled (`alluxio.user.metadata.cache.*`, off by default) — neither command reaches that one.

## Why It Happened

**Alluxio never watches your storage for changes.** There is no notification from S3 or HDFS telling Alluxio that a file changed, and Alluxio does not poll to find out. Once a file is cached, Alluxio keeps serving that copy.

How long it keeps serving it is decided by a **cache filter policy** — a rule you attach to a path that answers one question: *may Alluxio keep using its cached copy without re-checking the storage?*

You configure these rules in [Cache Policies](/ee-ai-en/ai-3.8-15.1.x/cache/cache-policies.md#controlling-what-to-cache-cache-filter-policies). There are three answers:

| Policy      | What it means                                       | When Alluxio re-checks your storage          |
| ----------- | --------------------------------------------------- | -------------------------------------------- |
| `immutable` | "This file never changes." **This is the default.** | Never                                        |
| `maxAge`    | "Trust the cached copy for N minutes/hours."        | On the first read after that time has passed |
| `skipCache` | "Don't cache this at all."                          | Every read                                   |

So if you never configured anything, your data is `immutable`: Alluxio assumes it will never change and never looks at the storage again.

That assumption is right for model weights and versioned datasets, and it is the fastest option. But when you *do* change the data underneath, nothing tells Alluxio — which is exactly the symptom at the top of this page.

Here is what each policy does when you change something in the storage:

| What you did in the storage | With `immutable` (default)                   | With `maxAge`                                     | With `skipCache` |
| --------------------------- | -------------------------------------------- | ------------------------------------------------- | ---------------- |
| Overwrote the file          | Reads return the old content                 | Old content until the age expires, then refreshed | Always current   |
| Changed its size            | Reads return the old content at the old size | Corrected after the age expires                   | Always current   |
| Deleted the file            | Reads still return the old content           | Corrected after the age expires                   | Always current   |

{% hint style="warning" %}
**These failures are silent.** A cached read of a file you deleted still returns `HTTP 200` and the old bytes — there is no error to alert you. It gets more confusing when tools disagree: after you replace a file with a larger one, `ls` reports the **new** size (it asks your storage) while a read still returns the **old, shorter** content from cache. If an application is reading data you changed underneath it, assume it is reading the old version until you invalidate. To make these cases raise an error instead of returning wrong data, see [Detecting Staleness on Read](#detecting-staleness-on-read).
{% endhint %}

`maxAge` has one detail worth knowing: the refresh happens **when someone reads the file**, not the moment the age expires. An expired entry that nobody touches just sits there until the next read.

## Preventing It Next Time

Running `job free` by hand works, but if the same path keeps going stale, the policy on it is wrong for how the data is actually used.

There are three postures, and they answer different questions. Most clusters end up combining them.

| Posture    | Mechanism                              | You find out a file went stale…            | Cost                                        |
| ---------- | -------------------------------------- | ------------------------------------------ | ------------------------------------------- |
| **React**  | `job free` on the path                 | when a person notices and runs it          | manual, one path at a time                  |
| **Bound**  | a `maxAge` policy                      | within the age you set                     | one revalidation per expiry; no failed read |
| **Detect** | `alluxio.snapshot.consistency.enabled` | on the next read that reaches your storage | one cut-off read per changed file           |

**One state defeats detection entirely: a fully cached file.** If every page is already in cache, the read never contacts your storage, so there is no version to compare against and you get a complete — but older — copy. No error, nothing truncated. Detection cannot see it; only `job free` (clears it now) or a `maxAge` policy (bounds how long it can last) will. This is why *Detect* is a complement to the other two rather than a replacement for them.

Pick the policy that matches how the data behaves:

| How your data behaves                                          | Use this                                                | Result                                                                                                                  |
| -------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Never changes once written (model weights, versioned datasets) | `immutable` (default)                                   | Fastest; no re-checks                                                                                                   |
| Updated on a schedule (daily ETL, retraining output)           | `maxAge` set to that interval                           | Refreshes itself; stale by at most one interval                                                                         |
| Changes constantly, or isn't worth caching                     | `skipCache`                                             | Always current; no cache benefit                                                                                        |
| Changes unpredictably, and you must not read stale data        | `skipCache`, or `maxAge` short enough to bound the risk | Trade-off between freshness and speed — see [below](#detecting-staleness-on-read) to get an error instead of wrong data |

{% hint style="warning" %}
Writing data through Alluxio does not need any of this — Alluxio updates its own cache on those writes. Policies matter only for data written **directly to your storage**, bypassing Alluxio.
{% endhint %}

### What About TTL?

They look interchangeable and are not. `maxAge` re-checks your storage on the next read after it expires; TTL just deletes the cached copy on a timer, without checking.

So: reach for TTL to stop data lingering in cache, never to keep reads current. See [Cache Policies](/ee-ai-en/ai-3.8-15.1.x/cache/cache-policies.md#difference-between-maxage-cache-filter-and-ttl-rules).

## Detecting Staleness on Read

The policies above decide *when* Alluxio re-checks your storage. This setting changes what happens when the data moves underneath a read that is already in flight:

```properties
alluxio.snapshot.consistency.enabled=true
```

Each read records the version of the object it is serving — on S3, the ETag — and checks every byte it returns against that version. If the object was replaced, the read is **cut off** rather than completed with a mix of old and new content.

How that surfaces matters, because it is not an HTTP error. The response has already started, so it keeps its `200` status and its declared `Content-Length` — 12582912 for the 12 MiB object below — and the body simply stops early.

The same read, of an object overwritten while one of its pages was cached:

|                  | Setting off                            | Setting on                                  |
| ---------------- | -------------------------------------- | ------------------------------------------- |
| Bytes delivered  | 12582912                               | 3145728                                     |
| Content          | first page old, rest new — **spliced** | old bytes only, then nothing                |
| `curl` exit code | `0`                                    | `18` (transfer closed with bytes remaining) |

Where it stops is not itself meaningful — it is however much had already reached the network, not a page or object boundary. What matters is that it stops short of the declared length.

So the setting does not remove the truncation — the truncation is *how* it avoids serving a splice.

**Any client that checks `Content-Length` against what it received sees the failure**: `curl` exits non-zero, an HTTP library raises an incomplete-read error, an S3 SDK raises a retryable transfer error.

A client that ignores transfer errors and keeps whatever it buffered ends up with a short file it believes is complete.

The interruption also repairs the cache: the worker drops the stale copy, so the next read returns the current object. Over ten consecutive reads of an overwritten file: one cut-off transfer, then nine correct ones, and never a spliced response.

One thing to know before enabling it: **one read is cut off per changed file.** That interruption *is* the detection, so it is expected rather than a fault.

That puts a requirement on your applications: one that retries on transfer errors sees a transient failure followed by correct data; one that treats any I/O error as fatal turns a self-healing event into a job failure.

{% hint style="info" %}
Requires AI-3.8-15.1.18 or later. Set it on the workers and the clients, and restart them to pick it up.
{% endhint %}

## Checking a Path's Policy

To see which rule is in effect on a path:

{% tabs %}
{% tab title="Kubernetes (Operator)" %}

```shell
kubectl exec -n <NAMESPACE> alluxio-cluster-coordinator-0 -- \
  alluxio fs ls -c <path>
```

{% endtab %}

{% tab title="Docker / Bare-Metal" %}

```shell
bin/alluxio fs ls -c <path>
```

{% endtab %}
{% endtabs %}

## Related Pages

* [Cache Policies](/ee-ai-en/ai-3.8-15.1.x/cache/cache-policies.md) — how to write and apply cache filter rules, TTL, pinning, quotas
* [Cache Eviction](/ee-ai-en/ai-3.8-15.1.x/cache/cache-eviction.md) — freeing cache space, and the complete `job free` reference
* [Metadata Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/metadata-listing.md) — directory-listing cache setup and tuning