Cache Insight

Cache Insight answers three questions about a running cluster: what is cached, where it is cached, and whether the under storage and the cache still agree. It performs a live, read-only scan of under-storage listings and worker cache metadata, and streams the result back as it goes.

It unifies capabilities that were previously spread across separate tools — single-file location lookup, cache-coverage checks over a file list, and worker page-store inventory — into one REST endpoint on the coordinator.

Cache Insight supersedes fs location and fs check-cached. Both of those commands are planned for deprecation; new automation should target the /api/v1/cache-insight endpoint.

Scans never block application reads or writes, never load data from under storage into the cache, and never mutate cache state. Because the cluster keeps serving traffic during a scan, results are approximate: a file can be cached, evicted, written, or deleted while the scan is in flight.

When to use it

Question	Request that answers it
Is this dataset warm enough to run the job?	"source":"ufs", "detail":"summary" over the dataset prefix
Which files (or byte ranges) still need prefetching?	"source":"ufs", "detail":"placement"
What is actually sitting in the cache right now?	"source":"worker", "detail":"summary"
Why is this one file reading slowly?	"detail":"full" on the explicit path
What would I lose by draining this worker?	"source":"worker" with a "workers" filter
Is the cache holding data that no longer exists in storage?	"source":"both" (look at the ORPHAN region)

Quick start

Send requests to the coordinator web port (default 19999). This scan covers one explicit file plus a recursive prefix, and prints the final coverage aggregate:

curl -sN -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/manifests/latest.json","s3://my-bucket/train/**"]}],
       "source":"ufs","detail":"summary"}' \
  | tail -1 | jq .

A browser-based view of the same API is served by the coordinator at /cache-insight-ui.

Path grammar

Paths use a trailing-glob grammar. Mid-path wildcards and extension patterns (*.parquet) are not supported.

Pattern	Scope
s3://bucket/dir/file	a single file
s3://bucket/dir/*	the directory, non-recursive (immediate children)
s3://bucket/dir/**	the directory, recursive
/**	the whole cluster namespace

A plain path that resolves to a directory is rejected — append /* or /** to state which one you mean. Overlapping paths within a request are also rejected, with the conflicting pairs named in the error.

Choosing a source

source selects which namespace is enumerated, and therefore which questions the scan is able to answer.

Source	Enumerates	Reads under storage	What it reveals	Use it for
ufs (default)	files in under storage	Yes	cache coverage for every listed file, against the candidate (consistent-hash) workers	read-hit coverage, prefetch planning
worker	resident cache entries	No	everything currently cached, including bytes sitting on non-candidate workers	cache inventory, placement, worker-drain analysis
both	under storage and cache	Yes	MISS / CACHED / ORPHAN regions, plus complete placement visibility	storage/cache reconciliation, garbage audits

Cache status always comes from the workers — under storage never knows what is cached. source chooses which side anchors the report rows.

Because ufs mode only asks the candidate workers, bytes parked on a worker that is not a placement candidate are invisible to it, and orphans cannot be detected at all. Use worker or both for misplacement and garbage questions.

Choosing a detail level

detail levels are strictly additive: each level keeps every field from the level below it and adds more placement structure. Pick the smallest level that answers your question — the RPC count is identical at every level, but the payload is not.

Detail	Per-file records	Adds	Recommended use
summary (default)	no	final aggregate only	dashboards, broad coverage checks
list	yes	status, coverage, age, per-copy rollups, segment tallies	inventories that need every file
placement	yes	missing/partial segment ranges, per-worker holders[]	prefetch and placement remediation
full	yes	the full segment × replica grid, plus file metadata for explicit paths	targeted single-file diagnosis

Following one partially cached file up the ladder:

// list — classification and compact copy summaries
{"record":"file","ufs_path":"s3://bucket/train/a.parquet","status":"PARTIAL","cached_pct":62,
 "copies":[{"replica_index":0,"complete":false,"cached_pct":62}]}

// placement — adds actionable segment ranges and worker-level placement
{"copies":[{"replica_index":0,"missing_segments":["2-4"]}],
 "holders":[{"worker_id":"worker-a","cached_pct":62,"cached_bytes":671088640}]}

// full — adds the segment-by-replica grid and, for explicit paths, file_info
{"segments":[{"id":0,"offset":0,"length":134217728,
              "replicas":[{"replica_index":0,"worker_id":"worker-a","cached_bytes":134217728}]}],
 "file_info":{"owner":"alluxio","group":"alluxio","mode":420}}

full payload grows with segments × replicas and is not capped, so a multi-TB file produces a very large record. Reserve full for explicit paths or small globs; placement carries the actionable ranges at a fraction of the size.

Large scans

A list scan emits one record per file — roughly 1 KB each — so 100,000 files produce about 100 MB of response and one million files about 1 GB. Two knobs keep that manageable:

• only: incomplete (experimental) emits only files that are not fully cached and consistent. The aggregate still folds every scanned file, so the totals stay exact while the stream carries just the actionable rows.

• sample: N scans a deterministic 1-in-N subset of enumerated files. Counts and byte totals become ×N estimates (relative error ≈ 1/√files) and the aggregate carries a sampled block. Use it for aggregates, not for inventories — unsampled files are simply not scanned. Explicit paths are never sampled, and re-runs pick the same files.

Reading the output

Each file is classified into one of six statuses, computed over the active candidate replicas (and, on a segmented cluster, derived from the per-segment rollup):

Status	Meaning
FULL_ALL_REPLICAS	fully cached on every expected replica
FULL_ONE_REPLICA	fully cached on at least one replica, but not all
PARTIAL	some bytes cached, no replica complete
NONE	no bytes cached on any candidate worker
FAILED	a candidate worker could not be evaluated
UNCHECKED	every candidate worker was excluded by the workers filter

In both mode each record additionally carries a region, describing which side of the join it came from:

Region	Meaning
MISS	listed in under storage, cached nowhere
CACHED	present on both sides
ORPHAN	resident in cache, absent from the current under-storage listing

The response is NDJSON (application/x-ndjson), not a single JSON document: one complete JSON value per line, streamed as the scan progresses.

header  →  file × N  →  heartbeat / progress / error …  →  aggregate (final)

Consume it without buffering the whole scan:

# Final aggregate only
curl -sN ... | tail -1 | jq .

# File records as they arrive
curl -sN ... | jq -c 'select(.record=="file")'

# Convert a bounded scan into one JSON array
curl -sN ... | jq -s .

Treat a captured result as complete only when its last record is an aggregate with "final": true. A stream that was disconnected — or a cancellation observed before its final aggregate — is incomplete. Parse line by line, and tolerate unknown fields and record types.

Proxies and web stacks may buffer streaming responses. When that happens, use GET /jobs/{id} for live progress and treat the stream purely as the result channel.

Common tasks

Sampled coverage dashboard — cheap enough to run on a schedule over the whole namespace:

curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/**"]}],"source":"ufs","sample":10}' \
  | tail -1 | jq '{counts, bytes, sampled}'

Prefetch planning — list the byte ranges that are still missing, in bulk:

curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/**"]}],"source":"ufs","detail":"placement"}' \
  | jq -r 'select(.record=="file" and .complete_copies==0)
           | [.ufs_path, (.copies[0].missing_segments // [] | join(","))] | @tsv'

Single-file forensics — the replacement for fs location:

curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/model.bin"]}],"source":"ufs","detail":"full"}' \
  | jq 'select(.record=="file") | {status, misplaced_bytes, segments}'

What would draining a worker lose? — restrict the scan to one worker, so every record's bytes are that worker's holdings:

curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/**"]}],"source":"worker","detail":"list",
       "workers":["worker-8cfadbfc-..."]}'

List the valid worker selectors with GET /api/v1/cache-insight/workers.

Explicit file list — batched point lookups, efficient at 10k+ paths:

jq -n --slurpfile p paths.json '{groups:[{paths:$p[0]}],source:"ufs",detail:"list"}' \
  | curl -sN -X POST http://coordinator:19999/api/v1/cache-insight \
      -H 'Content-Type: application/json' -d @-

Controlling scan cost

Every knob that shapes a scan is a per-request parameter, so an overloaded scan is fixed by cancelling and resubmitting — no configuration change and no restart.

Parameter	Default	Effect
groups[].limit	unlimited	stop after N files (or segments, under segmentation) and mark the aggregate truncated
scanRateLimit	100000	page-metadata visits per second, per worker, for this scan
workers	all	send RPCs only to the listed workers (id, host, or host:port)
sample	1 (exact)	scan a deterministic 1-in-N subset
batchSize	1000	files per worker RPC / listing flush / cursor page (range 1–10000)

Monitoring and cancelling long scans

Choose your own job_id at submit time, and progress and cancellation never depend on parsing the stream — useful behind buffering proxies:

# Submit with a client-chosen id
curl -s -X POST http://coordinator:19999/api/v1/cache-insight \
  -H 'Content-Type: application/json' \
  -d '{"groups":[{"paths":["s3://my-bucket/train/**"]}],"source":"both",
       "detail":"list","job_id":"audit-2026-07-14"}' &

# Poll it from any connection
curl -s http://coordinator:19999/api/v1/cache-insight/jobs/audit-2026-07-14

# Cancel it
curl -s -X DELETE http://coordinator:19999/api/v1/cache-insight/jobs/audit-2026-07-14

Every submission is logged on the coordinator as cache-insight scan <job_id> started, so "was it admitted?" is answerable from the log even if the response never arrived. Submitting a duplicate job_id while the first is still running returns 409 with that job's info, so a retry cannot accidentally double-run.

Cancellation is cooperative: DELETE returns 202 immediately, the scan unwinds at its next unit of work, and the stream ends with {"record":"aggregate","final":true,"aborted":true,"reason":"cancelled"}. Closing the HTTP connection cancels the scan too, within about one heartbeat interval.

Configuration

Only two properties, both hard operational limits:

Property	Default	Scope	Description
alluxio.coordinator.cache.insight.max.concurrent.scans	5	coordinator	Maximum concurrent scans. Requests beyond the cap get 429 plus the running-job list, so the caller can wait or cancel one.
alluxio.worker.cache.insight.scan.rate.limit	0 (off)	worker	Worker-global ceiling on page-metadata visit rate (pages/sec) across all in-flight Cache Insight RPCs on that worker, regardless of how many scans or callers compose. This is the hard brake; the per-request scanRateLimit still applies on top of it.

Metrics

Prometheus metrics are exposed on each process's standard metrics endpoint alongside the other alluxio_* series. Counters carry the exporter's _total suffix on the wire.

Coordinator — scan orchestration:

Metric	Type	Labels	Meaning
alluxio_cache_insight_scans	counter	state = started | completed | aborted	Scan lifecycle. started − completed − aborted ≈ in flight; a rising aborted rate means clients are cancelling or disconnecting mid-scan.
alluxio_cache_insight_active_scans	gauge	—	Scans running right now. Pinned at the admission cap means new requests are getting 429.
alluxio_cache_insight_files_checked	counter	—	Files examined across all scans and modes; its rate is scan throughput.
alluxio_cache_insight_fanout_rpcs	counter	result = success | failure	Per-worker RPCs issued by the coordinator fan-out. failure counts timeouts and errored workers; scans continue and affected files surface as error records.
alluxio_cache_insight_scan_duration_ms	histogram	source = ufs | worker | both	End-to-end scan duration; buckets 10 ms – 300 s.

Worker — scan work served:

Metric	Type	Labels	Meaning
alluxio_cache_insight_pages_visited	counter	—	Page-metadata entries visited by the scanner. This is the quantity alluxio.worker.cache.insight.scan.rate.limit bounds — compare its per-second rate against the configured limit to see whether the brake is engaged.
alluxio_cache_insight_worker_rpcs	counter	method = batch | scan	Cache Insight RPCs served: batch = point lookups for explicit paths and length probes, scan = cursor scans over the page metastore (worker and both modes).

Limitations

Results are approximate. A scan reads a live system without locks, so files cached, evicted, written, or deleted mid-scan can be counted on either side of the change. Treat the numbers as accurate to within concurrent activity; re-run to converge, and scan during quiet windows for audits.

Jobs live in coordinator memory. A coordinator restart forgets running scans, the stream dies with the connection, and there is no resume. Re-submit — scans are pure reads, so re-running is always safe. Workers act only on coordinator request, so an orphaned scan stops itself within one in-flight batch.

Results are not stored. The NDJSON stream is the only copy, and nothing is queryable once the scan ends. Persist the stream yourself (tee it to a file) if you need to re-analyze without re-scanning.

Cancellation is cooperative. DELETE returns 202 before the scan has fully stopped, so expect a short tail of records after cancelling. Wait for the aborted aggregate as the true end.

ufs mode has candidate-only visibility. Bytes on a worker that is not a placement candidate are invisible to it, and orphans are only detected in both mode. Use both or worker for misplacement and garbage audits.

full payload grows with segments × replicas. There is no cap on the per-file grid. Reserve full for explicit paths or small globs, and use placement otherwise.

Sampling trades per-file coverage for speed. With sample: N only the sampled files emit records, so use sampling for aggregates rather than inventories. The selection is deterministic, so re-runs cover the same files.

Security

The endpoint registers on the coordinator web server and inherits its authentication. It is read-only, but it is expensive and it reveals the namespace; admission control bounds concurrency, not access. Cache Insight has no per-user quotas — restrict access to the coordinator web port and apply authentication or authorization at your network or proxy layer.

Scans never write worker state. Metadata lookups that miss the worker metadata cache are answered from an under-storage read-through without persisting what they load, and those lookups are bounded: at most 16 concurrent length lookups per worker (256 cluster-wide), with the coordinator's own last-resort lookups running on a 16-thread pool.

Reference

REST API → Cache Insight documents every request field, record field, and status code.