> For the complete documentation index, see [llms.txt](https://documentation.alluxio.io/ee-ai-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.alluxio.io/ee-ai-en/ai-3.8-15.1.x/performance/fuse-write-cache.md). # FUSE Write Optimization {% hint style="warning" %} This feature is experimental since AI-3.8.15.1.4. {% endhint %} This guide shows how to use the [Write Cache](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md) backend through the FUSE POSIX interface, enabling low-latency writes via standard filesystem calls (`write()`, `open()`, `close()`). Write Cache must already be deployed before following this guide. ## How It Relates to S3-API Write Cache The Write Cache backend (FoundationDB metadata + NVMe data + async UFS persistence) is **shared** between the two access interfaces: | | [S3-API Write Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md) | FUSE Write Optimization (this guide) | | -------------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Interface** | S3-compatible API (`PUT`, `GET`) | POSIX filesystem calls (`write`, `read`) | | **Client** | AWS CLI, boto3, s3fs, any S3 client | Any POSIX application, ML frameworks, shell tools | | **Write policies** | `WRITE_THROUGH`, `WRITE_BACK`, `TRANSIENT` | Same | | **FoundationDB** | Required | Required (same FDB cluster) | | **Additional limitations** | None beyond S3 semantics | See [POSIX Compatibility in Write-Cache Mode](#posix-compatibility-in-write-cache-mode) | ## Before You Start * [ ] **Write Cache is already deployed** — FDB must be running and `alluxio.write.cache.enabled: "true"`. Verify: ```shell kubectl exec -n -coordinator-0 -- \ alluxio conf get alluxio.write.cache.enabled ``` Expected: `true`. If not, complete [S3-API Write Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md) first. * [ ] **FDB pods are healthy**: ```shell kubectl get pods -n | grep fdb ``` Expected: FDB controller, log, and storage pods all `Running`. * [ ] **FUSE PVC exists**: ```shell kubectl get pvc -n -fuse ``` Expected: PVC is present (status `Pending` before a pod mounts it is normal). ## Recommended Cluster Configuration When FUSE Write Cache is active, a larger fraction of NVMe capacity is consumed by unpersisted write data. Increase the pinned-space ratio from the default `0.3` to `0.5` in your `alluxio-cluster.yaml`: ```yaml spec: properties: alluxio.write.cache.enabled: "true" # Raise from default 0.3 to 0.5 to accommodate write-heavy FUSE workloads alluxio.worker.page.store.pinned.file.capacity.limit.ratio: "0.5" fdb: enabled: true ``` Apply the change: ```shell # Idempotent kubectl apply -f alluxio-cluster.yaml ``` > Size workers' NVMe accordingly: at ratio `0.5` and total cache `1 TiB`, up to 500 GiB can be occupied by unpersisted write data at any time. If incoming write throughput exceeds persistence throughput, space fills and Alluxio returns `out-of-space` errors. See [Cache Space Management](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md#cache-space-management). ## Deploy a FUSE Client Pod The operator creates a PVC named `-fuse` during cluster installation. Mount it with `mountPropagation: HostToContainer` for auto-recovery if the FUSE process restarts. ```yaml apiVersion: v1 kind: Pod metadata: name: fuse-test-0 namespace: spec: containers: - image: ubuntu:22.04 name: fuse-test command: ["sleep", "infinity"] volumeMounts: - mountPath: /data name: alluxio-pvc mountPropagation: HostToContainer volumes: - name: alluxio-pvc persistentVolumeClaim: claimName: -fuse ``` ```shell # Idempotent kubectl apply -f fuse-pod.yaml kubectl -n get pod fuse-test-0 ``` Expected: `STATUS = Running`, `READY = 1/1`. For full FUSE deployment options (DaemonSet, Docker / Bare-Metal), see [POSIX API](/ee-ai-en/ai-3.8-15.1.x/data-access/fuse-based-posix-api.md). ## Configure Write-Back Paths Write policies are configured at the path level — the same as in [S3-API Write Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md#path-level-configuration). The paths refer to the Alluxio namespace (e.g., `/s3/checkpoints`), not the FUSE mount path (`/data/s3/checkpoints`). Non-interactive configuration (for scripting): ```shell kubectl exec -i -n -coordinator-0 -- \ bash -c 'EDITOR="cp /dev/stdin" alluxio pathconfig edit' << 'EOF' { "apiVersion": "v1.0", "defaultRule": { "description": "Global default", "policyMode": "WRITE_THROUGH" }, "pathRules": [ { "alluxioPath": "/s3/checkpoints/**", "description": "Low-latency checkpoint writes", "policyMode": "WRITE_BACK", "properties": { "writeReplicas": 1 } } ] } EOF ``` Expected: `Update successful!` Verify a specific path resolves to the expected policy: ```shell kubectl exec -n -coordinator-0 -- \ alluxio pathconfig test --path /s3/checkpoints/epoch-1/model.pt ``` Expected: output contains `"policyMode": "WRITE_BACK"`. ## Verify Write-Back via FUSE Write a file through FUSE and confirm it is eventually persisted to UFS: ```shell # Write through FUSE kubectl exec -i -n fuse-test-0 -- \ bash -c 'echo "hello from fuse write cache" > /data/s3/checkpoints/test.txt' # Confirm visible in Alluxio namespace kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/test.txt ``` Wait for async persistence (up to `alluxio.write.cache.async.file.check.period`, default `10min`): ```shell # Poll until PERSISTED (8 iterations × 15 s = 2 minutes) for i in $(seq 1 8); do echo "--- Check $i/8 ---" NOT_PERSISTED=$(kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/ 2>&1 | grep -c "NOT_PERSISTED" || true) if [ "${NOT_PERSISTED}" = "0" ]; then echo "All files PERSISTED." break fi echo "${NOT_PERSISTED} file(s) still pending. Waiting 15s..." sleep 15 done ``` Expected: `All files PERSISTED.` within 2 minutes. *** ## POSIX Compatibility in Write-Cache Mode {% hint style="warning" %} The FUSE Write Cache is built on the same backend as the S3-API Write Cache. As a result, the FUSE mount has additional restrictions that go beyond both standard POSIX and standard FUSE semantics. These restrictions apply to **all write-cache policies** (`WRITE_THROUGH`, `WRITE_BACK`, `TRANSIENT`). Review them before migrating workloads to a write-cache FUSE mount. {% endhint %} ### rename() returns EIO All `rename()` operations fail with `EIO` while any write-cache policy is active, regardless of whether the file has been persisted to UFS. This affects: * Shell `mv` and `rename` * Python `os.rename()`, `pathlib.Path.rename()` * Write-then-rename patterns (write to `.tmp`, rename into place) **Workaround — silly rename interception (opt-in):** When applications perform `rm` on open files, Linux internally issues a `rename()` to `.fuse_hidden*`. Enable the interceptor to handle this transparently: ```yaml spec: properties: # CLIENT-scoped: set in FUSE client properties alluxio.fuse.silly.rename.interceptor.enabled: "true" ``` With this option enabled, Alluxio intercepts `.fuse_hidden*` renames and handles open-file deletion without triggering an S3 `CopyObject + DeleteObject`. Default is `false`. ### Files are write-once after close Once a file is closed, it cannot be re-opened for writing, appending, or truncating: | Operation | errno | | -------------------------------------------------------------------- | -------- | | `open(path, O_CREAT \| O_EXCL)` — file already exists | `EEXIST` | | `open(path, O_WRONLY)` or `open(path, O_RDWR)` — file already exists | `EACCES` | **Impact:** Applications that update files in place (databases, log rotation, config rewriters) will not work through a write-cache FUSE mount. The write-cache FUSE mount is best suited for write-once workloads: model checkpoints, training datasets, ETL stage outputs. ### Hard links are not supported `link()` returns `EOPNOTSUPP`. Tools that rely on hard links (`rsync --hard-links`, some package managers) will not work through the mount. ### Cache page reclaim on delete (15.1.3+ behavior) When a file is deleted via FUSE `rm` or `rm -rf`, cached pages are reclaimed on **all** workers that hold copies of the file — not only the hash-ring owner. In builds prior to 15.1.3, only the owner worker reclaimed pages; other workers retained orphaned pages until the next eviction cycle. *** ## Monitoring Async Persistence Two CLI commands (available 15.1.3+) let you inspect in-flight persist operations without waiting for `alluxio fs ls`: ```shell # List all files pending or in-progress on a specific worker kubectl exec -i -n -worker-0 -- \ alluxio async-persist list # Scope to a single worker by ID kubectl exec -i -n -worker-0 -- \ alluxio async-persist list --worker # Check the persist state and retry count for a specific path kubectl exec -i -n -coordinator-0 -- \ alluxio async-persist stat --path /s3/checkpoints/epoch-1/model.pt ``` Use `async-persist stat` when `alluxio fs ls` shows a file stuck in `NOT_PERSISTED` to determine whether the issue is in the queue or the upload itself. ## Key Configuration | Property | Default | Description | | ------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alluxio.write.cache.enabled` | `false` | Enables Write Cache (shared with S3 API). | | `alluxio.worker.page.store.pinned.file.capacity.limit.ratio` | `0.3` | Max fraction of NVMe capacity for unpersisted write data. Raise to `0.5` for write-heavy FUSE workloads. | | `alluxio.write.cache.async.file.check.period` | `10min` | Scan interval for orphan detection. Shorter values increase FDB load. | | `alluxio.write.cache.async.check.orphan.timeout` | `1h` | Uncommitted writes older than this are treated as abandoned and cleaned up. | | `alluxio.fuse.silly.rename.interceptor.enabled` | `false` | CLIENT-scoped. Intercepts `.fuse_hidden*` rename/unlink for transparent `rm` of open files. | | `alluxio.worker.mark.writing.files.duration` | `10min` | If a file is open for write but receives no new data for this duration, the worker treats it as a dangling write eligible for cleanup. Timer resets on every write. | ## Troubleshooting ### Directory deletion returns DEADLINE\_EXCEEDED Running `alluxio fs rm -R` or `rm -rf` on a `WRITE_BACK` path may fail with: ``` io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after ~5s ``` {% hint style="danger" %} Despite the error, the underlying files **may have already been deleted from UFS** before the timeout. Do not assume the data is still present. {% endhint %} **Recovery steps:** 1. Verify UFS state directly: ```shell aws s3 ls s3://// --recursive | head -20 ``` 2. If files are gone from S3, the deletion succeeded at the data layer. Re-running `alluxio fs rm -R` will confirm by returning `Path does not exist`. 3. Pagestore disk space may not shrink immediately — orphaned pages are reclaimed on the next eviction cycle. *** ### Files stuck in NOT\_PERSISTED ```shell kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/ ``` If files remain `NOT_PERSISTED` beyond `alluxio.write.cache.async.file.check.period`: 1. Check async-persist queue: ```shell kubectl exec -i -n -worker-0 -- \ alluxio async-persist list ``` 2. Check specific file status: ```shell kubectl exec -i -n -coordinator-0 -- \ alluxio async-persist stat --path /s3/checkpoints/ ``` 3. Check worker logs for upload errors: ```shell kubectl logs -n -worker-0 --tail=100 | \ grep -i "persist\|upload\|flush" ``` 4. If UFS is unreachable, retries enter exponential backoff (up to `alluxio.worker.write.cache.async.persist.retry.max.interval`, default `1h`). Verify UFS connectivity from the worker pod. *** ### rename() returns EIO unexpectedly This is expected behaviour when any write-cache policy is active (see [rename() returns EIO](#rename-returns-eio-unexpectedly)). If your application relies on rename: * Switch the affected path to `NO_CACHE` policy to bypass the write cache entirely for that path. * Enable `alluxio.fuse.silly.rename.interceptor.enabled: "true"` if the rename is triggered by `rm` of an open file. *** ### FUSE pod OOM or mount not connected These are not write-cache-specific. See [FUSE Troubleshooting](/ee-ai-en/ai-3.8-15.1.x/data-access/fuse-based-posix-api.md#troubleshooting). ## See Also * [S3-API Write Optimization](/ee-ai-en/ai-3.8-15.1.x/performance/s3-write-cache.md) — write cache via S3 API; deploy this first * [POSIX API](/ee-ai-en/ai-3.8-15.1.x/data-access/fuse-based-posix-api.md) — FUSE deployment details, mount options, read-cache mode * [S3 API Benchmarks](/ee-ai-en/ai-3.8-15.1.x/benchmark/s3-api.md) — S3-side write throughput baselines * [Benchmarking POSIX Performance](/ee-ai-en/ai-3.8-15.1.x/benchmark/benchmarking-posix-performance.md) — FUSE-side throughput baselines