FUSE Write Optimization

This feature is experimental since AI-3.8.15.1.6.

This guide shows how to use the Write Cache backend through the FUSE POSIX interface, enabling low-latency writes via standard filesystem calls (write(), open(), close()). Write Cache must already be deployed before following this guide.

How It Relates to S3-API Write Cache

The Write Cache backend (FoundationDB metadata + NVMe data + async UFS persistence) is shared between the two access interfaces:

	S3-API Write Optimization	FUSE Write Optimization (this guide)
Interface	S3-compatible API (PUT, GET)	POSIX filesystem calls (write, read)
Client	AWS CLI, boto3, s3fs, any S3 client	Any POSIX application, ML frameworks, shell tools
Write policies	WRITE_THROUGH, WRITE_BACK, TRANSIENT	Same
FoundationDB	Required	Required (same FDB cluster)
Additional limitations	None beyond S3 semantics	See POSIX Compatibility in Write-Cache Mode

Before You Start

• Write Cache is already deployed — FDB must be running and alluxio.write.cache.enabled: "true". Verify:

  kubectl exec -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
    alluxio conf get alluxio.write.cache.enabled

  Expected: true. If not, complete S3-API Write Optimization first.

• FDB pods are healthy:

  kubectl get pods -n <NAMESPACE> | grep fdb

  Expected: FDB controller, log, and storage pods all Running.

• FUSE PVC exists:

  kubectl get pvc -n <NAMESPACE> <CLUSTER_NAME>-fuse

  Expected: PVC is present (status Pending before a pod mounts it is normal).

Recommended Cluster Configuration

When FUSE Write Cache is active, a larger fraction of NVMe capacity is consumed by unpersisted write data. Increase the pinned-space ratio from the default 0.3 to 0.5 in your alluxio-cluster.yaml:

spec:
  properties:
    alluxio.write.cache.enabled: "true"
    # Raise from default 0.3 to 0.5 to accommodate write-heavy FUSE workloads
    alluxio.worker.page.store.pinned.file.capacity.limit.ratio: "0.5"
  fdb:
    enabled: true

Apply the change:

# Idempotent
kubectl apply -f alluxio-cluster.yaml

Size workers' NVMe accordingly: at ratio 0.5 and total cache 1 TiB, up to 500 GiB can be occupied by unpersisted write data at any time. If incoming write throughput exceeds persistence throughput, space fills and Alluxio returns out-of-space errors. See Cache Space Management.

Deploy a FUSE Client Pod

The operator creates a PVC named <CLUSTER_NAME>-fuse during cluster installation. Mount it with mountPropagation: HostToContainer for auto-recovery if the FUSE process restarts.

apiVersion: v1
kind: Pod
metadata:
  name: fuse-test-0
  namespace: <NAMESPACE>
spec:
  containers:
    - image: ubuntu:22.04
      name: fuse-test
      command: ["sleep", "infinity"]
      volumeMounts:
        - mountPath: /data
          name: alluxio-pvc
          mountPropagation: HostToContainer
  volumes:
    - name: alluxio-pvc
      persistentVolumeClaim:
        claimName: <CLUSTER_NAME>-fuse

# Idempotent
kubectl apply -f fuse-pod.yaml
kubectl -n <NAMESPACE> get pod fuse-test-0

Expected: STATUS = Running, READY = 1/1.

For full FUSE deployment options (DaemonSet, Docker / Bare-Metal), see POSIX API.

Configure Write-Back Paths

Write policies are configured at the path level — the same as in S3-API Write Optimization. The paths refer to the Alluxio namespace (e.g., /s3/checkpoints), not the FUSE mount path (/data/s3/checkpoints).

Non-interactive configuration (for scripting):

kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
  bash -c 'EDITOR="cp /dev/stdin" alluxio pathconfig edit' << 'EOF'
{
  "apiVersion": "v1.0",
  "defaultRule": {
    "description": "Global default",
    "policyMode": "WRITE_THROUGH"
  },
  "pathRules": [
    {
      "alluxioPath": "/s3/checkpoints/**",
      "description": "Low-latency checkpoint writes",
      "policyMode": "WRITE_BACK",
      "properties": { "writeReplicas": 1 }
    }
  ]
}
EOF

Expected: Update successful!

Verify a specific path resolves to the expected policy:

kubectl exec -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
  alluxio pathconfig test --path /s3/checkpoints/epoch-1/model.pt

Expected: output contains "policyMode": "WRITE_BACK".

Verify Write-Back via FUSE

Write a file through FUSE and confirm it is eventually persisted to UFS:

# Write through FUSE
kubectl exec -i -n <NAMESPACE> fuse-test-0 -- \
  bash -c 'echo "hello from fuse write cache" > /data/s3/checkpoints/test.txt'

# Confirm visible in Alluxio namespace
kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
  alluxio fs ls /s3/checkpoints/test.txt

Wait for async persistence (up to alluxio.write.cache.async.file.check.period, default 10min):

# Poll until PERSISTED (8 iterations × 15 s = 2 minutes)
for i in $(seq 1 8); do
  echo "--- Check $i/8 ---"
  NOT_PERSISTED=$(kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
    alluxio fs ls /s3/checkpoints/ 2>&1 | grep -c "NOT_PERSISTED" || true)
  if [ "${NOT_PERSISTED}" = "0" ]; then
    echo "All files PERSISTED."
    break
  fi
  echo "${NOT_PERSISTED} file(s) still pending. Waiting 15s..."
  sleep 15
done

Expected: All files PERSISTED. within 2 minutes.

---

POSIX Compatibility in Write-Cache Mode

The FUSE Write Cache is built on the same backend as the S3-API Write Cache. As a result, the FUSE mount has additional restrictions that go beyond both standard POSIX and standard FUSE semantics. These restrictions apply to all write-cache policies (WRITE_THROUGH, WRITE_BACK, TRANSIENT). Review them before migrating workloads to a write-cache FUSE mount.

The table below summarizes POSIX operation support when any write-cache policy (WRITE_THROUGH, WRITE_BACK, TRANSIENT) is active. Symbols: ✅ Supported | ⚠️ Limited | ❌ Not supported.

Category	Operation	Support	Notes
Read	read / readv / pread / preadv	✅
Write	Sequential write (new file)	✅	Core use case
	Overwrite existing file	✅	Via truncate + recreate
	Append existing file	⚠️	Architecture supports (MPU)
	fsync / fdatasync	✅	Triggers async persist
	truncate	✅	delete + recreate
	Concurrent writers (same file)	❌	S3 object model limitation
Rename	File rename	✅ Atomic	Works with any S3 bucket type; zero-copy (metadata-only)
	Directory rename	✅ Atomic	Critical for AI/ML checkpoint patterns
Delete	unlink	✅ Default enabled
	rm -rf	✅	Pages reclaimed on all workers (15.1.3+)
Directory	readdir	✅ Dual-path merge
	mkdir	✅ Persisted
	rmdir	✅ Empty directories
Metadata	stat / getattr	✅
	File size	✅
	mtime	✅	Read/write (utimens)
	atime / ctime	= mtime
	chmod / chown	❌	S3 has no POSIX permission model
	xattr	⚠️ Limited
Links	Symbolic link (symlink / readlink)	✅	Magic header encoding, up to 4096-byte target path
	Hard link (link())	❌ EOPNOTSUPP	S3 has no inode concept
Locking	flock (advisory)	✅ Single-node	libfuse only; no cross-node locking
	fcntl / lockf	❌

rename() support (15.1.6+)

Since AI-3.8.15.1.6, rename() is fully supported — both file and directory rename, atomic, with zero-copy metadata-only semantics (no S3 CopyObject + DeleteObject). This works with any S3 bucket type.

In builds prior to AI-3.8.15.1.6, all rename() operations failed with EIO, affecting:

• Shell mv and rename

• Python os.rename(), pathlib.Path.rename()

• Write-then-rename patterns (write to .tmp, rename into place)

Silly rename interception (opt-in): When applications perform rm on open files, Linux internally issues a rename() to .fuse_hidden*. Enable the interceptor to handle this transparently without triggering an S3 CopyObject + DeleteObject:

spec:
  properties:
    # CLIENT-scoped: set in FUSE client properties
    alluxio.fuse.silly.rename.interceptor.enabled: "true"

Default is false.

Overwrite via truncate + recreate (15.1.6+)

Since AI-3.8.15.1.6, overwriting an existing file is supported via truncate + recreate semantics: Alluxio deletes the existing file and creates a new one at the same path. Random in-place overwrites (pwrite) are not supported.

Prior to 15.1.6, files were write-once after close — re-opening an existing file for writing returned EACCES. If you are running an earlier build:

Operation	errno
open(path, O_CREAT | O_EXCL) — file already exists	EEXIST
open(path, O_WRONLY) or open(path, O_RDWR) — file already exists	EACCES

Impact: Applications that update files in place (databases, log rotation, config rewriters) are best served by AI-3.8.15.1.6+. The write-cache FUSE mount remains best suited for write-once workloads — model checkpoints, training datasets, ETL stage outputs — even with overwrite support.

Hard links are not supported

link() returns EOPNOTSUPP. Tools that rely on hard links (rsync --hard-links, some package managers) will not work through the mount.

Cache page reclaim on delete (15.1.3+ behavior)

When a file is deleted via FUSE rm or rm -rf, cached pages are reclaimed on all workers that hold copies of the file — not only the hash-ring owner. In builds prior to 15.1.3, only the owner worker reclaimed pages; other workers retained orphaned pages until the next eviction cycle.

Comparison with AWS Mountpoint for S3

Alluxio WriteCache (WRITE_BACK mode) provides a strict superset of the POSIX interface support offered by AWS Mountpoint for Amazon S3 — every POSIX operation that Mountpoint supports, Alluxio supports at an equal or higher level, with no exceptions.

Alluxio additionally provides the following POSIX capabilities that Mountpoint does not support:

• Atomic rename (file and directory, on any S3 bucket type)

• symlink / readlink (symbolic links)

• Persistent mkdir (directories survive restart)

• xattr (extended attributes)

Symbols: ✅ Supported | ⚠️ Limited / Conditional | ❌ Not supported.

Category	Operation	Alluxio WriteCache (WRITE_BACK mode)	AWS Mountpoint for S3	Notes
Read	read / readv / pread / preadv	✅	✅	Both fully support all read variants
Write	Sequential write (new file)	✅	✅	Core use case for both
	Random write (pwrite)	❌	❌	Neither supports random write in their primary write mode
	Overwrite existing file	✅ truncate + recreate	⚠️ --allow-overwrite + O_TRUNC	—
	Append existing file	⚠️ Architecture supports (MPU)	⚠️ S3 Express One Zone only	Both conditional
	Concurrent writers (same file)	❌	❌	S3 object model limitation
	fsync / fdatasync	✅ Triggers upload	⚠️ Commits to S3; no further writes	—
	truncate	✅ delete + recreate	⚠️ Requires --allow-overwrite	—
Rename	Single file rename	✅ Atomic	⚠️ S3 Express One Zone only	Alluxio works with any S3 bucket
	File rename with overwrite	✅	⚠️ S3 XOZ + --allow-overwrite	—
	Directory rename	✅ Atomic	❌	Alluxio advantage — critical for AI/ML checkpoint patterns
	Zero-copy rename	✅ Metadata-only, no data copy	❌ S3 rename = copy + delete	Key architectural advantage
Delete	unlink	✅ Default enabled	⚠️ Requires --allow-delete	—
	Recursive delete (rm -rf)	✅	✅ File-by-file	—
Directory	readdir	✅ Dual-path merge	✅	—
	mkdir	✅ Persisted	⚠️ Local only, not persisted to S3	Alluxio: true directory semantics
	rmdir	✅ Empty directories	⚠️ Only mkdir-created dirs	—
Metadata	stat / getattr	✅	✅ Limited	—
	File size	✅	✅	—
	mtime	✅ Read/write (utimens)	✅ Read-only	Alluxio supports write
	atime / ctime	= mtime	= mtime	—
	chmod / chown	❌	❌	S3 has no POSIX permission model
	xattr	⚠️ Limited	❌	—
Links	Symbolic link (symlink / readlink)	✅	❌	Alluxio: magic header encoding, up to 4096-byte target path
	Hard link	❌ EOPNOTSUPP	❌	S3 has no inode concept
Locking	flock (advisory)	✅ Single-node (libfuse)	✅ Single-node (libfuse)	Neither supports cross-node locking
	fcntl locking	❌	❌	—
	lockf	❌	❌	—
Other	mmap shared write	❌	❌	Requires page cache semantics
	Sparse files (file holes)	❌	❌	Conflicts with sequential write model

Consistency Model

Scenario	Alluxio WriteCache (WRITE_BACK mode)	AWS Mountpoint for S3
Read-after-write (local)	Immediately readable from write buffer after close	Readable after close (must wait for S3 upload to complete)
Visibility on S3	Async persist (seconds to minutes after close)	Immediately visible on S3 after close
Multi-instance consistency	Distributed metadata (strong consistency)	No cross-instance coordination
stat freshness	Strongly consistent	May lag by ~1 second under concurrent modification
Directory listing consistency	Strongly consistent	Strongly consistent (S3 ListObjects)
Durability on close	Data in worker NVMe cache (multi-replica); async to S3	Data on S3 (synchronous upload)
Node failure	Multi-replica protection — triggers early S3 persist	Unclosed writes are lost

Alluxio provides faster local read-after-write (no S3 round-trip) and strong cross-instance metadata consistency via its distributed metadata layer. Mountpoint provides stronger close-time S3 durability (synchronous upload). Alluxio compensates with multi-replica writes — if a worker fails, replicas on other workers preserve data and trigger early S3 persist.

---

Monitoring Async Persistence

Two CLI commands (available 15.1.3+) let you inspect in-flight persist operations without waiting for alluxio fs ls:

# List all files pending or in-progress on a specific worker
kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-worker-0 -- \
  alluxio async-persist list

# Scope to a single worker by ID
kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-worker-0 -- \
  alluxio async-persist list --worker <WORKER_ID>

# Check the persist state and retry count for a specific path
kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
  alluxio async-persist stat --path /s3/checkpoints/epoch-1/model.pt

Use async-persist stat when alluxio fs ls shows a file stuck in NOT_PERSISTED to determine whether the issue is in the queue or the upload itself.

Key Configuration

Property	Default	Description
alluxio.write.cache.enabled	false	Enables Write Cache (shared with S3 API).
alluxio.worker.page.store.pinned.file.capacity.limit.ratio	0.3	Max fraction of NVMe capacity for unpersisted write data. Raise to 0.5 for write-heavy FUSE workloads.
alluxio.write.cache.async.file.check.period	10min	Scan interval for orphan detection. Shorter values increase FDB load.
alluxio.write.cache.async.check.orphan.timeout	1h	Uncommitted writes older than this are treated as abandoned and cleaned up.
alluxio.fuse.silly.rename.interceptor.enabled	false	CLIENT-scoped. Intercepts .fuse_hidden* rename/unlink for transparent rm of open files.
alluxio.worker.mark.writing.files.duration	10min	If a file is open for write but receives no new data for this duration, the worker treats it as a dangling write eligible for cleanup. Timer resets on every write.

Troubleshooting

Directory deletion returns DEADLINE_EXCEEDED

Running alluxio fs rm -R or rm -rf on a WRITE_BACK path may fail with:

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after ~5s

Despite the error, the underlying files may have already been deleted from UFS before the timeout. Do not assume the data is still present.

Recovery steps:

1. Verify UFS state directly:

   aws s3 ls s3://<BUCKET>/<path>/ --recursive | head -20

2. If files are gone from S3, the deletion succeeded at the data layer. Re-running alluxio fs rm -R will confirm by returning Path does not exist.

3. Pagestore disk space may not shrink immediately — orphaned pages are reclaimed on the next eviction cycle.

---

Files stuck in NOT_PERSISTED

kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
  alluxio fs ls /s3/checkpoints/

If files remain NOT_PERSISTED beyond alluxio.write.cache.async.file.check.period:

1. Check async-persist queue:

   kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-worker-0 -- \
     alluxio async-persist list

2. Check specific file status:

   kubectl exec -i -n <NAMESPACE> <CLUSTER_NAME>-coordinator-0 -- \
     alluxio async-persist stat --path /s3/checkpoints/<filename>

3. Check worker logs for upload errors:

   kubectl logs -n <NAMESPACE> <CLUSTER_NAME>-worker-0 --tail=100 | \
     grep -i "persist\|upload\|flush"

4. If UFS is unreachable, retries enter exponential backoff (up to alluxio.worker.write.cache.async.persist.retry.max.interval, default 1h). Verify UFS connectivity from the worker pod.

---

rename() returns EIO unexpectedly

This is expected behaviour when any write-cache policy is active (see rename() returns EIO). If your application relies on rename:

• Switch the affected path to NO_CACHE policy to bypass the write cache entirely for that path.

• Enable alluxio.fuse.silly.rename.interceptor.enabled: "true" if the rename is triggered by rm of an open file.

---

FUSE pod OOM or mount not connected

These are not write-cache-specific. See FUSE Troubleshooting.

See Also

• S3-API Write Optimization — write cache via S3 API; deploy this first

• POSIX API — FUSE deployment details, mount options, read-cache mode

• S3 API Benchmarks — S3-side write throughput baselines

• Benchmarking POSIX Performance — FUSE-side throughput baselines