> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/cache/cache-insight.md).

# Cache Insight

Cache Insight answers three questions about a running cluster: **what is cached**, **where it is cached**, and **whether the under storage and the cache still agree**. It performs a live, read-only scan of under-storage listings and worker cache metadata, and streams the result back as it goes.

It unifies capabilities that were previously spread across separate tools — single-file location lookup, cache-coverage checks over a file list, and worker page-store inventory — into one REST endpoint on the coordinator.

{% hint style="info" %}
Cache Insight supersedes [`fs location`](/ee-ai-en/ai-3.8-15.1.x/reference/user-cli.md#fs-location) and [`fs check-cached`](/ee-ai-en/ai-3.8-15.1.x/reference/user-cli.md#fs-check-cached). Both of those commands are planned for deprecation; new automation should target the `/api/v1/cache-insight` endpoint.
{% endhint %}

Scans never block application reads or writes, never load data from under storage into the cache, and never mutate cache state. Because the cluster keeps serving traffic during a scan, results are **approximate**: a file can be cached, evicted, written, or deleted while the scan is in flight.

## When to use it

| Question                                                    | Request that answers it                                        |
| ----------------------------------------------------------- | -------------------------------------------------------------- |
| Is this dataset warm enough to run the job?                 | `"source":"ufs"`, `"detail":"summary"` over the dataset prefix |
| Which files (or byte ranges) still need prefetching?        | `"source":"ufs"`, `"detail":"placement"`                       |
| What is actually sitting in the cache right now?            | `"source":"worker"`, `"detail":"summary"`                      |
| Why is this one file reading slowly?                        | `"detail":"full"` on the explicit path                         |
| What would I lose by draining this worker?                  | `"source":"worker"` with a `"workers"` filter                  |
| Is the cache holding data that no longer exists in storage? | `"source":"both"` (look at the `ORPHAN` region)                |

## Quick start

Send requests to the coordinator web port (default `19999`). This scan covers one explicit file plus a recursive prefix, and prints the final coverage aggregate:

```shell
curl -sN -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/manifests/latest.json","s3://my-bucket/train/**"]}],
       "source":"ufs","detail":"summary"}' \
  | tail -1 | jq .
```

A browser-based view of the same API is served by the coordinator at `/cache-insight-ui`.

### Path grammar

Paths use a trailing-glob grammar. Mid-path wildcards and extension patterns (`*.parquet`) are not supported.

| Pattern                | Scope                                             |
| ---------------------- | ------------------------------------------------- |
| `s3://bucket/dir/file` | a single file                                     |
| `s3://bucket/dir/*`    | the directory, non-recursive (immediate children) |
| `s3://bucket/dir/**`   | the directory, recursive                          |
| `/**`                  | the whole cluster namespace                       |

A plain path that resolves to a directory is rejected — append `/*` or `/**` to state which one you mean. Overlapping paths within a request are also rejected, with the conflicting pairs named in the error.

## Choosing a source

`source` selects which namespace is enumerated, and therefore which questions the scan is able to answer.

| Source          | Enumerates                  | Reads under storage | What it reveals                                                                       | Use it for                                        |
| --------------- | --------------------------- | ------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------- |
| `ufs` (default) | files in under storage      | Yes                 | cache coverage for every listed file, against the candidate (consistent-hash) workers | read-hit coverage, prefetch planning              |
| `worker`        | resident cache entries      | No                  | everything currently cached, including bytes sitting on non-candidate workers         | cache inventory, placement, worker-drain analysis |
| `both`          | under storage **and** cache | Yes                 | `MISS` / `CACHED` / `ORPHAN` regions, plus complete placement visibility              | storage/cache reconciliation, garbage audits      |

Cache status always comes from the workers — under storage never knows what is cached. `source` chooses which side *anchors the report rows*.

Because `ufs` mode only asks the candidate workers, bytes parked on a worker that is not a placement candidate are invisible to it, and orphans cannot be detected at all. Use `worker` or `both` for misplacement and garbage questions.

## Choosing a detail level

`detail` levels are strictly additive: each level keeps every field from the level below it and adds more placement structure. Pick the smallest level that answers your question — the RPC count is identical at every level, but the payload is not.

| Detail              | Per-file records | Adds                                                                   | Recommended use                    |
| ------------------- | ---------------- | ---------------------------------------------------------------------- | ---------------------------------- |
| `summary` (default) | no               | final aggregate only                                                   | dashboards, broad coverage checks  |
| `list`              | yes              | status, coverage, age, per-copy rollups, segment tallies               | inventories that need every file   |
| `placement`         | yes              | missing/partial segment ranges, per-worker `holders[]`                 | prefetch and placement remediation |
| `full`              | yes              | the full segment × replica grid, plus file metadata for explicit paths | targeted single-file diagnosis     |

Following one partially cached file up the ladder:

```json
// list — classification and compact copy summaries
{"record":"file","ufs_path":"s3://bucket/train/a.parquet","status":"PARTIAL","cached_pct":62,
 "copies":[{"replica_index":0,"complete":false,"cached_pct":62}]}

// placement — adds actionable segment ranges and worker-level placement
{"copies":[{"replica_index":0,"missing_segments":["2-4"]}],
 "holders":[{"worker_id":"worker-a","cached_pct":62,"cached_bytes":671088640}]}

// full — adds the segment-by-replica grid and, for explicit paths, file_info
{"segments":[{"id":0,"offset":0,"length":134217728,
              "replicas":[{"replica_index":0,"worker_id":"worker-a","cached_bytes":134217728}]}],
 "file_info":{"owner":"alluxio","group":"alluxio","mode":420}}
```

{% hint style="warning" %}
`full` payload grows with segments × replicas and is not capped, so a multi-TB file produces a very large record. Reserve `full` for explicit paths or small globs; `placement` carries the actionable ranges at a fraction of the size.
{% endhint %}

### Large scans

A `list` scan emits one record per file — roughly 1 KB each — so 100,000 files produce about 100 MB of response and one million files about 1 GB. Two knobs keep that manageable:

* **`only: incomplete`** (experimental) emits only files that are not fully cached and consistent. The aggregate still folds every scanned file, so the totals stay exact while the stream carries just the actionable rows.
* **`sample: N`** scans a deterministic 1-in-N subset of enumerated files. Counts and byte totals become ×N estimates (relative error ≈ 1/√files) and the aggregate carries a `sampled` block. Use it for aggregates, not for inventories — unsampled files are simply not scanned. Explicit paths are never sampled, and re-runs pick the same files.

## Reading the output

Each file is classified into one of six statuses, computed over the active candidate replicas (and, on a segmented cluster, derived from the per-segment rollup):

| Status              | Meaning                                                     |
| ------------------- | ----------------------------------------------------------- |
| `FULL_ALL_REPLICAS` | fully cached on every expected replica                      |
| `FULL_ONE_REPLICA`  | fully cached on at least one replica, but not all           |
| `PARTIAL`           | some bytes cached, no replica complete                      |
| `NONE`              | no bytes cached on any candidate worker                     |
| `FAILED`            | a candidate worker could not be evaluated                   |
| `UNCHECKED`         | every candidate worker was excluded by the `workers` filter |

In `both` mode each record additionally carries a `region`, describing which side of the join it came from:

| Region   | Meaning                                                          |
| -------- | ---------------------------------------------------------------- |
| `MISS`   | listed in under storage, cached nowhere                          |
| `CACHED` | present on both sides                                            |
| `ORPHAN` | resident in cache, absent from the current under-storage listing |

The response is **NDJSON** (`application/x-ndjson`), not a single JSON document: one complete JSON value per line, streamed as the scan progresses.

```
header  →  file × N  →  heartbeat / progress / error …  →  aggregate (final)
```

Consume it without buffering the whole scan:

```shell
# Final aggregate only
curl -sN ... | tail -1 | jq .

# File records as they arrive
curl -sN ... | jq -c 'select(.record=="file")'

# Convert a bounded scan into one JSON array
curl -sN ... | jq -s .
```

{% hint style="info" %}
Treat a captured result as complete only when its last record is an `aggregate` with `"final": true`. A stream that was disconnected — or a cancellation observed before its final aggregate — is incomplete. Parse line by line, and tolerate unknown fields and record types.
{% endhint %}

Proxies and web stacks may buffer streaming responses. When that happens, use `GET /jobs/{id}` for live progress and treat the stream purely as the result channel.

## Common tasks

**Sampled coverage dashboard** — cheap enough to run on a schedule over the whole namespace:

```shell
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/**"]}],"source":"ufs","sample":10}' \
  | tail -1 | jq '{counts, bytes, sampled}'
```

**Prefetch planning** — list the byte ranges that are still missing, in bulk:

```shell
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/**"]}],"source":"ufs","detail":"placement"}' \
  | jq -r 'select(.record=="file" and .complete_copies==0)
           | [.ufs_path, (.copies[0].missing_segments // [] | join(","))] | @tsv'
```

**Single-file forensics** — the replacement for `fs location`:

```shell
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/model.bin"]}],"source":"ufs","detail":"full"}' \
  | jq 'select(.record=="file") | {status, misplaced_bytes, segments}'
```

**What would draining a worker lose?** — restrict the scan to one worker, so every record's bytes are that worker's holdings:

```shell
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/**"]}],"source":"worker","detail":"list",
       "workers":["worker-8cfadbfc-..."]}'
```

List the valid worker selectors with `GET /api/v1/cache-insight/workers`.

**Explicit file list** — batched point lookups, efficient at 10k+ paths:

```shell
jq -n --slurpfile p paths.json '{groups:[{paths:$p[0]}],source:"ufs",detail:"list"}' \
  | curl -sN -X POST http://coordinator:19999/api/v1/cache-insight \
      -H 'Content-Type: application/json' -d @-
```

## Controlling scan cost

Every knob that shapes a scan is a per-request parameter, so an overloaded scan is fixed by cancelling and resubmitting — no configuration change and no restart.

| Parameter        | Default     | Effect                                                                                  |
| ---------------- | ----------- | --------------------------------------------------------------------------------------- |
| `groups[].limit` | unlimited   | stop after N files (or segments, under segmentation) and mark the aggregate `truncated` |
| `scanRateLimit`  | 100000      | page-metadata visits per second, per worker, for this scan                              |
| `workers`        | all         | send RPCs only to the listed workers (id, host, or `host:port`)                         |
| `sample`         | `1` (exact) | scan a deterministic 1-in-N subset                                                      |
| `batchSize`      | 1000        | files per worker RPC / listing flush / cursor page (range 1–10000)                      |

## Monitoring and cancelling long scans

Choose your own `job_id` at submit time, and progress and cancellation never depend on parsing the stream — useful behind buffering proxies:

```shell
# Submit with a client-chosen id
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/**"]}],"source":"both",
       "detail":"list","job_id":"audit-2026-07-14"}' &

# Poll it from any connection
curl -s http://coordinator:19999/api/v1/cache-insight/jobs/audit-2026-07-14

# Cancel it
curl -s -X DELETE http://coordinator:19999/api/v1/cache-insight/jobs/audit-2026-07-14
```

Every submission is logged on the coordinator as `cache-insight scan <job_id> started`, so "was it admitted?" is answerable from the log even if the response never arrived. Submitting a duplicate `job_id` while the first is still running returns `409` with that job's info, so a retry cannot accidentally double-run.

Cancellation is cooperative: `DELETE` returns `202` immediately, the scan unwinds at its next unit of work, and the stream ends with `{"record":"aggregate","final":true,"aborted":true,"reason":"cancelled"}`. Closing the HTTP connection cancels the scan too, within about one heartbeat interval.

## Configuration

Only two properties, both hard operational limits:

| Property                                                 | Default   | Scope       | Description                                                                                                                                                                                                                                                |
| -------------------------------------------------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio.coordinator.cache.insight.max.concurrent.scans` | `5`       | coordinator | Maximum concurrent scans. Requests beyond the cap get `429` plus the running-job list, so the caller can wait or cancel one.                                                                                                                               |
| `alluxio.worker.cache.insight.scan.rate.limit`           | `0` (off) | worker      | Worker-global ceiling on page-metadata visit rate (pages/sec) across all in-flight Cache Insight RPCs on that worker, regardless of how many scans or callers compose. This is the hard brake; the per-request `scanRateLimit` still applies on top of it. |

## Metrics

Prometheus metrics are exposed on each process's standard metrics endpoint alongside the other `alluxio_*` series. Counters carry the exporter's `_total` suffix on the wire.

Coordinator — scan orchestration:

| Metric                                   | Type      | Labels                                          | Meaning                                                                                                                                                         |
| ---------------------------------------- | --------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alluxio_cache_insight_scans`            | counter   | `state` = `started` \| `completed` \| `aborted` | Scan lifecycle. `started − completed − aborted` ≈ in flight; a rising `aborted` rate means clients are cancelling or disconnecting mid-scan.                    |
| `alluxio_cache_insight_active_scans`     | gauge     | —                                               | Scans running right now. Pinned at the admission cap means new requests are getting `429`.                                                                      |
| `alluxio_cache_insight_files_checked`    | counter   | —                                               | Files examined across all scans and modes; its rate is scan throughput.                                                                                         |
| `alluxio_cache_insight_fanout_rpcs`      | counter   | `result` = `success` \| `failure`               | Per-worker RPCs issued by the coordinator fan-out. `failure` counts timeouts and errored workers; scans continue and affected files surface as `error` records. |
| `alluxio_cache_insight_scan_duration_ms` | histogram | `source` = `ufs` \| `worker` \| `both`          | End-to-end scan duration; buckets 10 ms – 300 s.                                                                                                                |

Worker — scan work served:

| Metric                                | Type    | Labels                       | Meaning                                                                                                                                                                                                                  |
| ------------------------------------- | ------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `alluxio_cache_insight_pages_visited` | counter | —                            | Page-metadata entries visited by the scanner. This is the quantity `alluxio.worker.cache.insight.scan.rate.limit` bounds — compare its per-second rate against the configured limit to see whether the brake is engaged. |
| `alluxio_cache_insight_worker_rpcs`   | counter | `method` = `batch` \| `scan` | Cache Insight RPCs served: `batch` = point lookups for explicit paths and length probes, `scan` = cursor scans over the page metastore (`worker` and `both` modes).                                                      |

## Limitations

**Results are approximate.** A scan reads a live system without locks, so files cached, evicted, written, or deleted mid-scan can be counted on either side of the change. Treat the numbers as accurate to within concurrent activity; re-run to converge, and scan during quiet windows for audits.

**Jobs live in coordinator memory.** A coordinator restart forgets running scans, the stream dies with the connection, and there is no resume. Re-submit — scans are pure reads, so re-running is always safe. Workers act only on coordinator request, so an orphaned scan stops itself within one in-flight batch.

**Results are not stored.** The NDJSON stream is the only copy, and nothing is queryable once the scan ends. Persist the stream yourself (`tee` it to a file) if you need to re-analyze without re-scanning.

**Cancellation is cooperative.** `DELETE` returns `202` before the scan has fully stopped, so expect a short tail of records after cancelling. Wait for the `aborted` aggregate as the true end.

**`ufs` mode has candidate-only visibility.** Bytes on a worker that is not a placement candidate are invisible to it, and orphans are only detected in `both` mode. Use `both` or `worker` for misplacement and garbage audits.

**`full` payload grows with segments × replicas.** There is no cap on the per-file grid. Reserve `full` for explicit paths or small globs, and use `placement` otherwise.

**Sampling trades per-file coverage for speed.** With `sample: N` only the sampled files emit records, so use sampling for aggregates rather than inventories. The selection is deterministic, so re-runs cover the same files.

## Security

The endpoint registers on the coordinator web server and inherits its authentication. It is read-only, but it is expensive and it reveals the namespace; admission control bounds concurrency, not access. Cache Insight has no per-user quotas — restrict access to the coordinator web port and apply authentication or authorization at your network or proxy layer.

Scans never write worker state. Metadata lookups that miss the worker metadata cache are answered from an under-storage read-through without persisting what they load, and those lookups are bounded: at most 16 concurrent length lookups per worker (256 cluster-wide), with the coordinator's own last-resort lookups running on a 16-thread pool.

## Reference

[REST API → Cache Insight](/ee-ai-en/ai-3.8-15.1.x/reference/rest-api.md#cache-insight) documents every request field, record field, and status code.