# FUSE Write Optimization {% hint style="warning" %} This feature is experimental since AI-3.8.15.1.6. {% endhint %} This guide shows how to use the [Write Cache](/ee-ai-en/performance/s3-write-cache.md) backend through the FUSE POSIX interface, enabling low-latency writes via standard filesystem calls (`write()`, `open()`, `close()`). Write Cache must already be deployed before following this guide. ## How It Relates to S3-API Write Cache The Write Cache backend (FoundationDB metadata + NVMe data + async UFS persistence) is **shared** between the two access interfaces: | | [S3-API Write Optimization](/ee-ai-en/performance/s3-write-cache.md) | FUSE Write Optimization (this guide) | | -------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Interface** | S3-compatible API (`PUT`, `GET`) | POSIX filesystem calls (`write`, `read`) | | **Client** | AWS CLI, boto3, s3fs, any S3 client | Any POSIX application, ML frameworks, shell tools | | **Write policies** | `WRITE_THROUGH`, `WRITE_BACK`, `TRANSIENT` | Same | | **FoundationDB** | Required | Required (same FDB cluster) | | **Additional limitations** | None beyond S3 semantics | See [POSIX Compatibility in Write-Cache Mode](#posix-compatibility-in-write-cache-mode) | ## Before You Start * [ ] **Write Cache is already deployed** — FDB must be running and `alluxio.write.cache.enabled: "true"`. Verify: ```shell kubectl exec -n -coordinator-0 -- \ alluxio conf get alluxio.write.cache.enabled ``` Expected: `true`. If not, complete [S3-API Write Optimization](/ee-ai-en/performance/s3-write-cache.md) first. * [ ] **FDB pods are healthy**: ```shell kubectl get pods -n | grep fdb ``` Expected: FDB controller, log, and storage pods all `Running`. * [ ] **FUSE PVC exists**: ```shell kubectl get pvc -n -fuse ``` Expected: PVC is present (status `Pending` before a pod mounts it is normal). ## Recommended Cluster Configuration When FUSE Write Cache is active, a larger fraction of NVMe capacity is consumed by unpersisted write data. Increase the pinned-space ratio from the default `0.3` to `0.5` in your `alluxio-cluster.yaml`: ```yaml spec: properties: alluxio.write.cache.enabled: "true" # Raise from default 0.3 to 0.5 to accommodate write-heavy FUSE workloads alluxio.worker.page.store.pinned.file.capacity.limit.ratio: "0.5" fdb: enabled: true ``` Apply the change: ```shell # Idempotent kubectl apply -f alluxio-cluster.yaml ``` > Size workers' NVMe accordingly: at ratio `0.5` and total cache `1 TiB`, up to 500 GiB can be occupied by unpersisted write data at any time. If incoming write throughput exceeds persistence throughput, space fills and Alluxio returns `out-of-space` errors. See [Cache Space Management](/ee-ai-en/performance/s3-write-cache.md#cache-space-management). ## Deploy a FUSE Client Pod The operator creates a PVC named `-fuse` during cluster installation. Mount it with `mountPropagation: HostToContainer` for auto-recovery if the FUSE process restarts. ```yaml apiVersion: v1 kind: Pod metadata: name: fuse-test-0 namespace: spec: containers: - image: ubuntu:22.04 name: fuse-test command: ["sleep", "infinity"] volumeMounts: - mountPath: /data name: alluxio-pvc mountPropagation: HostToContainer volumes: - name: alluxio-pvc persistentVolumeClaim: claimName: -fuse ``` ```shell # Idempotent kubectl apply -f fuse-pod.yaml kubectl -n get pod fuse-test-0 ``` Expected: `STATUS = Running`, `READY = 1/1`. For full FUSE deployment options (DaemonSet, Docker / Bare-Metal), see [POSIX API](/ee-ai-en/data-access/fuse-based-posix-api.md). ## Configure Write-Back Paths Write policies are configured at the path level — the same as in [S3-API Write Optimization](/ee-ai-en/performance/s3-write-cache.md#path-level-configuration). The paths refer to the Alluxio namespace (e.g., `/s3/checkpoints`), not the FUSE mount path (`/data/s3/checkpoints`). Non-interactive configuration (for scripting): ```shell kubectl exec -i -n -coordinator-0 -- \ bash -c 'EDITOR="cp /dev/stdin" alluxio pathconfig edit' << 'EOF' { "apiVersion": "v1.0", "defaultRule": { "description": "Global default", "policyMode": "WRITE_THROUGH" }, "pathRules": [ { "alluxioPath": "/s3/checkpoints/**", "description": "Low-latency checkpoint writes", "policyMode": "WRITE_BACK", "properties": { "writeReplicas": 1 } } ] } EOF ``` Expected: `Update successful!` Verify a specific path resolves to the expected policy: ```shell kubectl exec -n -coordinator-0 -- \ alluxio pathconfig test --path /s3/checkpoints/epoch-1/model.pt ``` Expected: output contains `"policyMode": "WRITE_BACK"`. ## Verify Write-Back via FUSE Write a file through FUSE and confirm it is eventually persisted to UFS: ```shell # Write through FUSE kubectl exec -i -n fuse-test-0 -- \ bash -c 'echo "hello from fuse write cache" > /data/s3/checkpoints/test.txt' # Confirm visible in Alluxio namespace kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/test.txt ``` Wait for async persistence (up to `alluxio.write.cache.async.file.check.period`, default `10min`): ```shell # Poll until PERSISTED (8 iterations × 15 s = 2 minutes) for i in $(seq 1 8); do echo "--- Check $i/8 ---" NOT_PERSISTED=$(kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/ 2>&1 | grep -c "NOT_PERSISTED" || true) if [ "${NOT_PERSISTED}" = "0" ]; then echo "All files PERSISTED." break fi echo "${NOT_PERSISTED} file(s) still pending. Waiting 15s..." sleep 15 done ``` Expected: `All files PERSISTED.` within 2 minutes. *** ## POSIX Compatibility in Write-Cache Mode {% hint style="warning" %} The FUSE Write Cache is built on the same backend as the S3-API Write Cache. As a result, the FUSE mount has additional restrictions that go beyond both standard POSIX and standard FUSE semantics. These restrictions apply to **all write-cache policies** (`WRITE_THROUGH`, `WRITE_BACK`, `TRANSIENT`). Review them before migrating workloads to a write-cache FUSE mount. {% endhint %} The table below summarizes POSIX operation support when any write-cache policy (`WRITE_THROUGH`, `WRITE_BACK`, `TRANSIENT`) is active. Symbols: ✅ Supported | ⚠️ Limited | ❌ Not supported. | Category | Operation | Support | Notes | | ------------- | -------------------------------------- | ------------------- | -------------------------------------------------------- | | **Read** | `read` / `readv` / `pread` / `preadv` | ✅ | | | **Write** | Sequential write (new file) | ✅ | Core use case | | | Overwrite existing file | ✅ | Via truncate + recreate | | | Append existing file | ⚠️ | Architecture supports (MPU) | | | `fsync` / `fdatasync` | ✅ | Triggers async persist | | | `truncate` | ✅ | delete + recreate | | | Concurrent writers (same file) | ❌ | S3 object model limitation | | **Rename** | File rename | ✅ Atomic | Works with any S3 bucket type; zero-copy (metadata-only) | | | Directory rename | ✅ Atomic | Critical for AI/ML checkpoint patterns | | **Delete** | `unlink` | ✅ Default enabled | | | | `rm -rf` | ✅ | Pages reclaimed on all workers (15.1.3+) | | **Directory** | `readdir` | ✅ Dual-path merge | | | | `mkdir` | ✅ Persisted | | | | `rmdir` | ✅ Empty directories | | | **Metadata** | `stat` / `getattr` | ✅ | | | | File size | ✅ | | | | `mtime` | ✅ | Read/write (`utimens`) | | | `atime` / `ctime` | = `mtime` | | | | `chmod` / `chown` | ❌ | S3 has no POSIX permission model | | | `xattr` | ⚠️ Limited | | | **Links** | Symbolic link (`symlink` / `readlink`) | ✅ | Magic header encoding, up to 4096-byte target path | | | Hard link (`link()`) | ❌ `EOPNOTSUPP` | S3 has no inode concept | | **Locking** | `flock` (advisory) | ✅ Single-node | libfuse only; no cross-node locking | | | `fcntl` / `lockf` | ❌ | | ### rename() support (15.1.6+) Since AI-3.8.15.1.6, `rename()` is fully supported — both file and directory rename, atomic, with zero-copy metadata-only semantics (no S3 `CopyObject + DeleteObject`). This works with any S3 bucket type. In builds **prior to AI-3.8.15.1.6**, all `rename()` operations failed with `EIO`, affecting: * Shell `mv` and `rename` * Python `os.rename()`, `pathlib.Path.rename()` * Write-then-rename patterns (write to `.tmp`, rename into place) **Silly rename interception (opt-in):** When applications perform `rm` on open files, Linux internally issues a `rename()` to `.fuse_hidden*`. Enable the interceptor to handle this transparently without triggering an S3 `CopyObject + DeleteObject`: ```yaml spec: properties: # CLIENT-scoped: set in FUSE client properties alluxio.fuse.silly.rename.interceptor.enabled: "true" ``` Default is `false`. ### Overwrite via truncate + recreate (15.1.6+) Since AI-3.8.15.1.6, overwriting an existing file is supported via `truncate` + recreate semantics: Alluxio deletes the existing file and creates a new one at the same path. Random in-place overwrites (`pwrite`) are not supported. Prior to 15.1.6, files were write-once after close — re-opening an existing file for writing returned `EACCES`. If you are running an earlier build: | Operation | errno | | -------------------------------------------------------------------- | -------- | | `open(path, O_CREAT \| O_EXCL)` — file already exists | `EEXIST` | | `open(path, O_WRONLY)` or `open(path, O_RDWR)` — file already exists | `EACCES` | **Impact:** Applications that update files in place (databases, log rotation, config rewriters) are best served by AI-3.8.15.1.6+. The write-cache FUSE mount remains best suited for write-once workloads — model checkpoints, training datasets, ETL stage outputs — even with overwrite support. ### Hard links are not supported `link()` returns `EOPNOTSUPP`. Tools that rely on hard links (`rsync --hard-links`, some package managers) will not work through the mount. ### Cache page reclaim on delete (15.1.3+ behavior) When a file is deleted via FUSE `rm` or `rm -rf`, cached pages are reclaimed on **all** workers that hold copies of the file — not only the hash-ring owner. In builds prior to 15.1.3, only the owner worker reclaimed pages; other workers retained orphaned pages until the next eviction cycle. ### Comparison with AWS Mountpoint for S3 Alluxio WriteCache (WRITE\_BACK mode) provides a **strict superset** of the POSIX interface support offered by AWS Mountpoint for Amazon S3 — every POSIX operation that Mountpoint supports, Alluxio supports at an equal or higher level, with no exceptions. Alluxio additionally provides the following POSIX capabilities that Mountpoint does not support: * Atomic `rename` (file and directory, on any S3 bucket type) * `symlink` / `readlink` (symbolic links) * Persistent `mkdir` (directories survive restart) * `xattr` (extended attributes) Symbols: ✅ Supported | ⚠️ Limited / Conditional | ❌ Not supported. | Category | Operation | Alluxio WriteCache (WRITE\_BACK mode) | AWS Mountpoint for S3 | Notes | | ------------- | -------------------------------------- | ------------------------------------- | ----------------------------------- | --------------------------------------------------------------- | | **Read** | `read` / `readv` / `pread` / `preadv` | ✅ | ✅ | Both fully support all read variants | | **Write** | Sequential write (new file) | ✅ | ✅ | Core use case for both | | | Random write (`pwrite`) | ❌ | ❌ | Neither supports random write in their primary write mode | | | Overwrite existing file | ✅ truncate + recreate | ⚠️ `--allow-overwrite` + `O_TRUNC` | — | | | Append existing file | ⚠️ Architecture supports (MPU) | ⚠️ S3 Express One Zone only | Both conditional | | | Concurrent writers (same file) | ❌ | ❌ | S3 object model limitation | | | `fsync` / `fdatasync` | ✅ Triggers upload | ⚠️ Commits to S3; no further writes | — | | | `truncate` | ✅ delete + recreate | ⚠️ Requires `--allow-overwrite` | — | | **Rename** | Single file rename | ✅ Atomic | ⚠️ S3 Express One Zone only | **Alluxio works with any S3 bucket** | | | File rename with overwrite | ✅ | ⚠️ S3 XOZ + `--allow-overwrite` | — | | | Directory rename | ✅ Atomic | ❌ | **Alluxio advantage** — critical for AI/ML checkpoint patterns | | | Zero-copy rename | ✅ Metadata-only, no data copy | ❌ S3 rename = copy + delete | Key architectural advantage | | **Delete** | `unlink` | ✅ Default enabled | ⚠️ Requires `--allow-delete` | — | | | Recursive delete (`rm -rf`) | ✅ | ✅ File-by-file | — | | **Directory** | `readdir` | ✅ Dual-path merge | ✅ | — | | | `mkdir` | ✅ Persisted | ⚠️ Local only, not persisted to S3 | **Alluxio**: true directory semantics | | | `rmdir` | ✅ Empty directories | ⚠️ Only `mkdir`-created dirs | — | | **Metadata** | `stat` / `getattr` | ✅ | ✅ Limited | — | | | File size | ✅ | ✅ | — | | | `mtime` | ✅ Read/write (`utimens`) | ✅ Read-only | Alluxio supports write | | | `atime` / `ctime` | = `mtime` | = `mtime` | — | | | `chmod` / `chown` | ❌ | ❌ | S3 has no POSIX permission model | | | `xattr` | ⚠️ Limited | ❌ | — | | **Links** | Symbolic link (`symlink` / `readlink`) | ✅ | ❌ | **Alluxio**: magic header encoding, up to 4096-byte target path | | | Hard link | ❌ `EOPNOTSUPP` | ❌ | S3 has no inode concept | | **Locking** | `flock` (advisory) | ✅ Single-node (libfuse) | ✅ Single-node (libfuse) | Neither supports cross-node locking | | | `fcntl` locking | ❌ | ❌ | — | | | `lockf` | ❌ | ❌ | — | | **Other** | `mmap` shared write | ❌ | ❌ | Requires page cache semantics | | | Sparse files (file holes) | ❌ | ❌ | Conflicts with sequential write model | **Consistency Model** | Scenario | Alluxio WriteCache (WRITE\_BACK mode) | AWS Mountpoint for S3 | | ----------------------------- | -------------------------------------------------------- | ---------------------------------------------------------- | | Read-after-write (local) | **Immediately readable** from write buffer after close | Readable after close (must wait for S3 upload to complete) | | Visibility on S3 | Async persist (seconds to minutes after close) | Immediately visible on S3 after close | | Multi-instance consistency | **Distributed metadata** (strong consistency) | No cross-instance coordination | | `stat` freshness | **Strongly consistent** | May lag by \~1 second under concurrent modification | | Directory listing consistency | **Strongly consistent** | Strongly consistent (S3 ListObjects) | | Durability on close | Data in worker NVMe cache (multi-replica); async to S3 | Data on S3 (synchronous upload) | | Node failure | **Multi-replica protection** — triggers early S3 persist | Unclosed writes are lost | Alluxio provides faster local read-after-write (no S3 round-trip) and strong cross-instance metadata consistency via its distributed metadata layer. Mountpoint provides stronger close-time S3 durability (synchronous upload). Alluxio compensates with multi-replica writes — if a worker fails, replicas on other workers preserve data and trigger early S3 persist. *** ## Monitoring Async Persistence Two CLI commands (available 15.1.3+) let you inspect in-flight persist operations without waiting for `alluxio fs ls`: ```shell # List all files pending or in-progress on a specific worker kubectl exec -i -n -worker-0 -- \ alluxio async-persist list # Scope to a single worker by ID kubectl exec -i -n -worker-0 -- \ alluxio async-persist list --worker # Check the persist state and retry count for a specific path kubectl exec -i -n -coordinator-0 -- \ alluxio async-persist stat --path /s3/checkpoints/epoch-1/model.pt ``` Use `async-persist stat` when `alluxio fs ls` shows a file stuck in `NOT_PERSISTED` to determine whether the issue is in the queue or the upload itself. ## Key Configuration | Property | Default | Description | | ------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alluxio.write.cache.enabled` | `false` | Enables Write Cache (shared with S3 API). | | `alluxio.worker.page.store.pinned.file.capacity.limit.ratio` | `0.3` | Max fraction of NVMe capacity for unpersisted write data. Raise to `0.5` for write-heavy FUSE workloads. | | `alluxio.write.cache.async.file.check.period` | `10min` | Scan interval for orphan detection. Shorter values increase FDB load. | | `alluxio.write.cache.async.check.orphan.timeout` | `1h` | Uncommitted writes older than this are treated as abandoned and cleaned up. | | `alluxio.fuse.silly.rename.interceptor.enabled` | `false` | CLIENT-scoped. Intercepts `.fuse_hidden*` rename/unlink for transparent `rm` of open files. | | `alluxio.worker.mark.writing.files.duration` | `10min` | If a file is open for write but receives no new data for this duration, the worker treats it as a dangling write eligible for cleanup. Timer resets on every write. | ## Troubleshooting ### Directory deletion returns DEADLINE\_EXCEEDED Running `alluxio fs rm -R` or `rm -rf` on a `WRITE_BACK` path may fail with: ``` io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after ~5s ``` {% hint style="danger" %} Despite the error, the underlying files **may have already been deleted from UFS** before the timeout. Do not assume the data is still present. {% endhint %} **Recovery steps:** 1. Verify UFS state directly: ```shell aws s3 ls s3://// --recursive | head -20 ``` 2. If files are gone from S3, the deletion succeeded at the data layer. Re-running `alluxio fs rm -R` will confirm by returning `Path does not exist`. 3. Pagestore disk space may not shrink immediately — orphaned pages are reclaimed on the next eviction cycle. *** ### Files stuck in NOT\_PERSISTED ```shell kubectl exec -i -n -coordinator-0 -- \ alluxio fs ls /s3/checkpoints/ ``` If files remain `NOT_PERSISTED` beyond `alluxio.write.cache.async.file.check.period`: 1. Check async-persist queue: ```shell kubectl exec -i -n -worker-0 -- \ alluxio async-persist list ``` 2. Check specific file status: ```shell kubectl exec -i -n -coordinator-0 -- \ alluxio async-persist stat --path /s3/checkpoints/ ``` 3. Check worker logs for upload errors: ```shell kubectl logs -n -worker-0 --tail=100 | \ grep -i "persist\|upload\|flush" ``` 4. If UFS is unreachable, retries enter exponential backoff (up to `alluxio.worker.write.cache.async.persist.retry.max.interval`, default `1h`). Verify UFS connectivity from the worker pod. *** ### rename() returns EIO unexpectedly This is expected behaviour when any write-cache policy is active (see [rename() returns EIO](#rename-returns-eio)). If your application relies on rename: * Switch the affected path to `NO_CACHE` policy to bypass the write cache entirely for that path. * Enable `alluxio.fuse.silly.rename.interceptor.enabled: "true"` if the rename is triggered by `rm` of an open file. *** ### FUSE pod OOM or mount not connected These are not write-cache-specific. See [FUSE Troubleshooting](/ee-ai-en/data-access/fuse-based-posix-api.md#troubleshooting). ## See Also * [S3-API Write Optimization](/ee-ai-en/performance/s3-write-cache.md) — write cache via S3 API; deploy this first * [POSIX API](/ee-ai-en/data-access/fuse-based-posix-api.md) — FUSE deployment details, mount options, read-cache mode * [S3 API Benchmarks](/ee-ai-en/benchmark/s3-api.md) — S3-side write throughput baselines * [Benchmarking POSIX Performance](/ee-ai-en/benchmark/benchmarking-posix-performance.md) — FUSE-side throughput baselines --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://documentation.alluxio.io/ee-ai-en/performance/fuse-write-cache.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.