Setup & Configuration

This page covers the base setup for Alluxio's S3-compatible API, including endpoint configuration, authentication, load balancing, and client compatibility. This setup applies to both the standard read-cache mode and the Write Cache mode.

Looking for Write Cache? If your workload requires low-latency writes with async persistence, complete the setup on this page first, then continue to Write Cache for FoundationDB deployment and write policy configuration.

Architecture Overview

┌─────────────────────────────────┐
│         S3 Clients              │
│  (AWS CLI, boto3, PyTorch, ...) │
└──────────┬──────────────────────┘
           │  S3 API (HTTP/HTTPS)
           ▼
┌─────────────────────────────────┐
│   Load Balancer (NLB/Nginx)     │
└──────────┬──────────────────────┘
           │  Uniformly distributed across workers
           ▼
┌─────────────────────────────────────────────────┐
│  Alluxio Workers (S3 API on port 29998)         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │ Worker 1 │  │ Worker 2 │  │ Worker N │ ...   │
│  │ NVMe SSD │  │ NVMe SSD │  │ NVMe SSD │       │
│  └──────────┘  └──────────┘  └──────────┘       │
└──────────┬──────────────────────────────────────┘
           │  Cache miss → fetch from UFS
           ▼
┌─────────────────────────────────────┐
│   Underlying Storage (S3/HDFS/...)  │
└─────────────────────────────────────┘

• Each Alluxio worker exposes an S3-compatible endpoint and manages a local NVMe/SSD cache.

• The load balancer distributes client requests across workers. With HTTP 307 redirects enabled, the receiving worker redirects the client to the data-owning worker for zero-copy reads.

• Both data and metadata are distributed across workers using consistent hashing. Each worker owns a portion of the namespace and manages both the cached data and the metadata for its assigned hash range. On a cache miss, the owning worker fetches the data from the underlying storage (UFS) and caches it locally.

Getting Started

1. Enable the S3 API

Enable the following property on all Alluxio workers:

alluxio.worker.s3.api.enabled=true

The S3 API is exposed on every Alluxio worker on the following ports:

• HTTP: 29998 (default)

• HTTPS: 29996 (default, requires TLS configuration)

2. (Recommended) Set Up a Load Balancer

It is highly recommended to place a load balancer in front of all workers to distribute S3 requests evenly. The load balancer address becomes the S3 endpoint for your clients.

Kubernetes (AWS NLB)

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  properties:
    alluxio.worker.s3.api.enabled: "true"
  worker:
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Other environments: Use Nginx, HAProxy, or DNS round-robin to distribute requests across worker nodes on port 29998.

3. Configure Your S3 Client

Authentication

Alluxio's S3 API supports two authentication methods:

SIMPLE (Default) — Alluxio parses the AWS Signature V4 Authorization header to extract the username, but does not validate the signature.

• Access Key: The Alluxio username to perform operations as. If omitted, operations run as the user that launched the worker process.

• Secret Key: Any non-empty value. Required by the client to generate the signature, but ignored by Alluxio.

OIDC — For centralized identity management using OpenID Connect tokens, refer to the Authentication guide.

Addressing Style

Clients must use path-style requests (http://<endpoint>/<bucket>/<object>). Virtual-hosted style requests are not supported.

# Configure AWS CLI to use path-style
aws configure set default.s3.addressing_style path

4. Verify the Connection

Use the AWS CLI to confirm the S3 endpoint is reachable:

# List all buckets (Alluxio mount points)
aws s3 ls --endpoint-url http://<LOAD_BALANCER_ADDRESS>

# Upload a test file
aws s3 cp test.txt s3://<bucket>/test.txt --endpoint-url http://<LOAD_BALANCER_ADDRESS>

# Download it back
aws s3 cp s3://<bucket>/test.txt downloaded.txt --endpoint-url http://<LOAD_BALANCER_ADDRESS>

Performance Tuning

Enabling HTTP Redirects

By default, Alluxio uses HTTP 307 redirects for optimal read performance. The flow works as follows:

1. Client sends a GET request to any worker (typically via the load balancer).

2. The receiving worker looks up which worker owns the cached data based on consistent hashing.

3. If the data is on a different worker, it returns an HTTP 307 redirect pointing to the data owner.

4. The client follows the redirect and reads directly from the data-owning worker — zero-copy, no proxy overhead.

This is why redirects are enabled by default: they eliminate an extra network hop and allow reads to scale linearly with the number of workers. However, not all S3 clients handle HTTP 307 redirects correctly. The table below summarizes compatibility:

Client	Supports HTTP 307 Redirect	Action Required
AWS CLI	Yes	None
COSBench	Yes	None
boto3 (Python)	No	Disable redirects
PyTorch S3 Connector	No	Disable redirects
MinIO Warp	No	Disable redirects

For clients that do not support redirects, disable them:

alluxio.worker.s3.redirect.enabled=false

When redirects are disabled, data is proxied through the worker that receives the request, which introduces an extra network hop. For latency-sensitive or throughput-sensitive workloads, prefer clients that support redirects or connect clients directly to individual workers.

Setting Key Alluxio Parameters

The following parameters control S3 API performance. Apply the recommended values for high-throughput workloads.

• alluxio.worker.s3.connection.keep.alive.enabled — Default: false; Recommended: true. Reuses TCP connections across requests to reduce handshake overhead and improve throughput under high concurrency.

• alluxio.worker.s3.connection.idle.max.time — Default: 0sec. Idle timeout for keep-alive connections; 0 means no timeout.

• alluxio.worker.s3.access.logging.enabled — Default: false; Recommended: true for production. When false, only failed requests are logged; when true, every request is logged. Useful for auditing and debugging; disable during benchmarks to avoid I/O overhead.

• alluxio.worker.s3.only.https.access — Default: false. When enabled, rejects non-HTTPS requests.

• alluxio.worker.s3.redirect.health.check.enabled — Default: true; Recommended: false for benchmarks. Disables per-request RPC health checks before redirect, reducing overhead and improving throughput; keep enabled in production for safety.

Linux Kernel Parameters

For intensive S3 benchmarks with high TCP connection rates, tuning kernel parameters can improve connection reuse and lower latency.

sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_fin_timeout=30

# Only on older kernels (e.g., CentOS 7 / kernel 3.10):
sysctl -w net.ipv4.tcp_tw_recycle=1

Parameter	Effect
net.ipv4.tcp_tw_reuse	Reuses sockets in TIME_WAIT state, preventing port exhaustion during high-frequency requests.
net.ipv4.tcp_tw_recycle	Accelerates TIME_WAIT cleanup. Removed in Linux 4.12+ — this parameter only exists on older kernels (e.g., CentOS 7 with kernel 3.10). On newer kernels, this sysctl key does not exist and the command will fail.
net.ipv4.tcp_fin_timeout	Reduces idle connection close time, freeing resources faster. Default: 60s → recommended: 30s.

Warning: Modifying kernel parameters can impact system stability. Ensure you understand these settings before applying them, especially in production environments.

NAT Compatibility: On older kernels where tcp_tw_recycle is available, enabling it may cause connection failures for clients behind a NAT device due to timestamp validation issues. Do not use in NAT environments.

Advanced Features

MultiPartUpload (MPU)

For large object uploads that pass through to the underlying storage, these parameters control multipart upload behavior:

Parameter	Default	Recommended	Description
alluxio.underfs.object.store.multipart.upload.async	false	true	Enables async multipart uploads to UFS, improving write throughput.
alluxio.underfs.s3.upload.threads.max	20	256 for high-throughput	Maximum concurrent upload threads per worker for S3 UFS writes.
alluxio.underfs.s3.multipart.upload.buffer.number	64	256 for high-throughput	Number of multipart upload buffers. Increase alongside upload.threads.max.

By default, multipart uploads are pass-through — each part is uploaded directly to the underlying storage, and the object is committed there upon CompleteMultipartUpload. Alluxio does not buffer the parts locally.

If you need to write parts to Alluxio's cache layer first and asynchronously persist them to the underlying storage, see S3 Write Cache.

Tagging and Metadata

• Enable Tagging: Requires extended attribute support for your UFS:

  alluxio.underfs.xattr.change.enabled=true

• Tag Limits: User-defined tags are limited to 10 per object/bucket per S3 tag restrictions. Disable with alluxio.worker.s3.tagging.restrictions.enabled=false.

• Metadata Size: User-defined metadata is limited to 2KB per S3 metadata restrictions. Adjust with alluxio.worker.s3.header.metadata.max.size.

Limitations

Before adopting Alluxio's S3 API, be aware of these constraints:

• Path-style only — Virtual-hosted style requests (http://<bucket>.<endpoint>/...) are not supported. All clients must use path-style (http://<endpoint>/<bucket>/<key>).

• Buckets: Only top-level directories in the Alluxio namespace are treated as S3 buckets. The root directory (/) is not a bucket. To preserve existing S3 URIs, use the bucket name as the mount path:

  alluxio mount add --path /<bucket-name> --ufs-uri s3://<bucket-name>/

• No Versioning or Locking: If multiple clients write to the same object simultaneously, the last write wins.

• Unsupported Characters: Do not use ?, \, ./, or ../ in object keys. Using // may lead to undefined behavior.

• Folder Objects: Subdirectories are returned as 0-byte folder objects in ListObjects(V2) responses, matching AWS S3 console behavior.

• No Atomicity: If-Match and If-None-Match headers are not supported in GetObject and PutObject.

• Signatures are not validated — Alluxio extracts the username from the Authorization header but does not verify the cryptographic signature. The secret key can be any non-empty string.

• Some clients break with default redirects — Clients like boto3 and aws sdk cannot follow HTTP 307 redirects to non-AWS endpoints . You must set alluxio.worker.s3.redirect.enabled=false before using them. See HTTP Redirects and Client Compatibility. Alternatively, you may use minio-py which supports HTTP 307 redirection.

• No virtual-hosted DNS — There is no DNS wildcard resolution for <bucket>.endpoint; this is a corollary of path-style only.

Supported S3 Actions

The following table lists supported S3 API actions. For detailed usage, see the official S3 API documentation.

S3 API Action	Supported Headers	Supported Query Parameters
AbortMultipartUpload	N/A	N/A
CompleteMultipartUpload	N/A	N/A
CopyObject	Content-Type, x-amz-copy-source, x-amz-metadata-directive, x-amz-tagging-directive, x-amz-tagging	N/A
CreateMultipartUpload	N/A	N/A
DeleteBucketTagging	N/A	N/A
DeleteObject	N/A	N/A
DeleteObjects	N/A	N/A
DeleteObjectTagging	N/A	N/A
GetBucketTagging	N/A	N/A
GetObject	Range	N/A
GetObjectTagging	N/A	N/A
HeadBucket	N/A	N/A
HeadObject	N/A	N/A
ListBuckets	N/A	N/A
ListMultipartUploads	N/A	N/A
ListObjects	N/A	delimiter, encoding-type, marker, max-keys, prefix
ListObjectsV2	N/A	continuation-token, delimiter, encoding-type, max-keys, prefix, start-after
ListParts	N/A	N/A
PutBucketTagging	N/A	N/A
PutObject	Content-Length, Content-MD5, Content-Type, x-amz-tagging	N/A
PutObjectTagging	N/A	N/A
UploadPart	Content-Length, Content-MD5	N/A

Usage Examples

Note: For clients that do not support HTTP 307 redirects (boto3, PyTorch), you must set alluxio.worker.s3.redirect.enabled=false before using them. See HTTP Redirects and Client Compatibility for details.

boto3

import boto3

s3 = boto3.client(
    "s3",
    aws_access_key_id="placeholder",      # Alluxio username (or any value)
    aws_secret_access_key="placeholder",   # Ignored by Alluxio
    region_name="us-east-1",
    endpoint_url="http://<LOAD_BALANCER_ADDRESS>"
    # Or connect directly: "http://<ALLUXIO_WORKER>:29998"
)

# List buckets (Alluxio mount points)
response = s3.list_buckets()
for bucket in response.get("Buckets", []):
    print(f" - {bucket['Name']}")

Install with: pip install boto3

PyTorch S3 Connector

from s3torchconnector import S3IterableDataset, S3ClientConfig

s3_client_config = S3ClientConfig(force_path_style=True)

dataset = S3IterableDataset.from_prefix(
    "s3://s3-mount",                                   # Alluxio mount point
    region="us-east-1",
    endpoint="http://<LOAD_BALANCER_ADDRESS>",
    s3client_config=s3_client_config,
)

for item in dataset:
    content = item.read()
    print(f"{item.key}: {len(content)} bytes")

Install with:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install s3torchconnector

See Also

• S3 Write Cache — low-latency writes with async persistence (requires FoundationDB)

• Benchmarking S3 API Performance — COSBench/Warp setup, performance baselines, and Linux kernel tuning

• S3 UFS Integration — multipart upload tuning, high concurrency settings, and S3 region configuration