S3 API

Alluxio exposes an S3-compatible REST API that allows applications built for Amazon S3 to read and write data without code changes. This page covers endpoint configuration, authentication, load balancing, and client compatibility.

Need low-latency writes? If your workload requires millisecond-level PUT latency or async persistence, see S3-API Write Optimization.

Prerequisites

Complete Installing on Kubernetes through Step 5 (Mount Storage) before proceeding. The S3 API requires:

A running Alluxio cluster (CLUSTERPHASE = Ready)
At least one UFS mount point — mount points become the S3 buckets exposed by the API

Quick Start

This section gets you to a working S3 endpoint in three steps, configured for worker direct mode — the mode every S3 client works with, including AWS CLI, boto3, and PyTorch S3 Connector. This is not the shipped default: worker redirect mode is, and AWS SDK-based clients cannot use it, so Step 1 turns redirect off explicitly. See Deployment Modes for the full comparison.

Step 1: Enable the S3 API

Add the following to spec.properties in alluxio-cluster.yaml and apply:

spec:
  properties:
    alluxio.worker.s3.api.enabled: "true"
    # Redirect mode is the default; turn it off so AWS SDK-based clients work.
    alluxio.worker.s3.redirect.enabled: "false"

kubectl apply -f alluxio-cluster.yaml

The S3 API is exposed on every worker on port 29998 (HTTP) and 29996 (HTTPS, requires TLS).

Step 2: Expose the Endpoint

With a load balancer (recommended for production)

Add a worker service to alluxio-cluster.yaml:

spec:
  worker:
    service:
      type: LoadBalancer

kubectl apply -f alluxio-cluster.yaml

Wait for the external IP to be assigned:

kubectl -n alx-ns get svc alluxio-cluster-worker --watch

✅ Success: EXTERNAL-IP is populated. Use that address as <ENDPOINT> below.

Without a load balancer (eval/single-node)

Use port-forward to reach a worker directly:

kubectl -n alx-ns port-forward pod/alluxio-cluster-worker-0 29998:29998 &

Use http://localhost:29998 as <ENDPOINT>.

Step 3: Verify with AWS CLI

Configure path-style requests (required — virtual-hosted style is not supported):

aws configure set default.s3.addressing_style path

With SIMPLE auth (the default), any non-empty credentials are accepted:

# List buckets (Alluxio mount points)
AWS_ACCESS_KEY_ID=testuser AWS_SECRET_ACCESS_KEY=testpassword \
  aws s3 ls --endpoint-url http://<ENDPOINT>

# Upload a test file
AWS_ACCESS_KEY_ID=testuser AWS_SECRET_ACCESS_KEY=testpassword \
  aws s3 cp test.txt s3://<bucket>/test.txt --endpoint-url http://<ENDPOINT>

# Download it back
AWS_ACCESS_KEY_ID=testuser AWS_SECRET_ACCESS_KEY=testpassword \
  aws s3 cp s3://<bucket>/test.txt downloaded.txt --endpoint-url http://<ENDPOINT>

Quick boto3 example:

import boto3

s3 = boto3.client(
    "s3",
    aws_access_key_id="placeholder",
    aws_secret_access_key="placeholder",
    region_name="us-east-1",
    endpoint_url="http://<ENDPOINT>",
)

for bucket in s3.list_buckets().get("Buckets", []):
    print(bucket["Name"])

Want higher throughput? The worker direct mode configured above traverses the cluster network twice for cross-worker reads, roughly halving small-object throughput compared to redirect mode. If your client follows HTTP 307 redirects to non-AWS endpoints, see worker redirect mode — the shipped default — for near-linear scaling.

Deployment Modes

Every Alluxio worker exposes an S3 endpoint on port 29998 and owns a slice of the namespace via consistent hashing. When a request lands on a worker that does not own the requested data, that data still has to reach the client. How it gets there defines three deployment modes:

Worker redirect mode (alluxio.worker.s3.redirect.enabled=true, the default) — the receiving worker returns an HTTP 307 so the client reconnects straight to the data owner. Only for clients that follow 307 redirects to non-AWS endpoints.
Worker direct mode (alluxio.worker.s3.redirect.enabled=false) — the client talks to a worker directly; a non-owner worker fetches the data from the owner and relays it back. Works with every client, including AWS SDK-based ones.
Client proxy mode (redirect.enabled=true + the node-local S3 Proxy) — a proxy next to the client resolves the owner from the hash ring and forwards there in a single hop. Gives AWS SDK clients near-redirect throughput.

AWS SDK clients need an explicit setting. Redirect mode is the default, but the AWS SDK does not follow HTTP 307 redirects to non-AWS endpoints. If your clients are AWS CLI, boto3, or the PyTorch S3 Connector, either set alluxio.worker.s3.redirect.enabled=false (worker direct mode) or deploy client proxy mode.

Background: what HTTP 307 means here. HTTP 307 is a redirect status code — the server responds with a new URL instead of data. The client then opens a fresh connection directly to that URL. Critically, no data payload traverses the original worker: the redirect response is a tiny HTTP message, and the full read happens worker-to-client without a proxy intermediary. The only cost is one extra round-trip for the redirect itself.

When to Use Which Mode

Your client

Recommended mode

Why

Follows HTTP 307 to non-AWS endpoints (COSBench, minio-py)

Worker redirect mode (default)

Client connects straight to the data owner — maximum, linearly-scaling throughput. AWS SDK clients cannot use it.

AWS SDK / boto3 / AWS CLI / PyTorch S3 Connector — simplest setup, no extra components

Worker direct mode

Works with every client, no code changes; needs alluxio.worker.s3.redirect.enabled=false. Cross-worker reads take an extra hop (~50% throughput vs. redirect for small objects).

AWS SDK / boto3 / AWS CLI / PyTorch S3 Connector — needs near-redirect throughput

Client proxy mode

A node-local proxy routes each request to the owner in one hop, so AWS SDK clients get near-redirect throughput. Adds one component to run.

For large sequential reads (e.g. 1 GiB+ model shards) the redirect handshake is amortized to near zero, so worker direct mode and worker redirect mode perform within a few percent of each other; the ~50% gap is a small-object effect. See Benchmarking S3 API Performance.

Worker redirect mode

Best for: COSBench, minio-py, and any client that correctly follows HTTP 307 redirects to non-AWS endpoints. This is the default (alluxio.worker.s3.redirect.enabled=true): the receiving worker checks ownership and issues a 307 only when needed — if the request already lands on the data owner, no redirect occurs. The client connects directly to the data-owning worker — zero-copy, no proxy overhead — and read throughput scales linearly with the number of workers.

Constraint: the AWS SDK does not follow 307 redirects to non-AWS endpoints, so clients built on it (AWS CLI, boto3, PyTorch S3 Connector) cannot use this mode — use worker direct mode or client proxy mode instead.

┌─────────────────────────────────┐
│         S3 Clients              │
│      (minio-py, COSBench)       │
└──────────┬──────────────────────┘
           │  S3 API (HTTP)
           ▼
┌─────────────────────────────────┐
│   Load Balancer (NLB/Nginx)     │
└──────────┬──────────────────────┘
           │  Distributed across workers
           ▼
┌─────────────────────────────────────────────────┐
│  Alluxio Workers (port 29998)                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │ Worker 1 │  │ Worker 2 │  │ Worker N │ ...   │
│  │ NVMe SSD │  │ NVMe SSD │  │ NVMe SSD │       │
│  └──────────┘  └──────────┘  └──────────┘       │
└──────────┬──────────────────────────────────────┘
           │  Cache miss → fetch from UFS
           ▼
┌─────────────────────────────────────┐
│   Underlying Storage (S3/HDFS/...)  │
└─────────────────────────────────────┘

Kubernetes (Operator)

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  properties:
    alluxio.worker.s3.api.enabled: "true"
    alluxio.worker.s3.redirect.enabled: "true"
  worker:
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

kubectl apply -f alluxio-cluster.yaml

Wait for the NLB to be provisioned (typically 1–2 minutes on AWS):

kubectl -n alx-ns get svc alluxio-cluster-worker --watch

✅ Success: EXTERNAL-IP is populated. Use that hostname as the S3 endpoint.

Docker / Bare-Metal

Use Nginx, HAProxy, or DNS round-robin to distribute requests across worker nodes on port 29998.

Worker direct mode

Set alluxio.worker.s3.redirect.enabled=false — redirect mode is the default, so worker direct mode has to be turned on explicitly; this is what the Quick Start configures. Clients connect to a worker — directly or through a load balancer over the workers on port 29998 — and that worker serves the request; when it is not the data owner it fetches the data from the owner and relays it back. Every client works with no code changes.

Trade-off: a cross-worker read traverses the cluster network twice (data owner → receiving worker → client), roughly halving aggregate throughput versus worker redirect mode for small objects.

Point your S3 endpoint at a load balancer over the workers:

http://<LOAD_BALANCER_ADDRESS>:29998

Single-worker deployments: the extra hop only happens when the receiving worker is not the data owner. With a single worker that never happens, so worker direct mode and worker redirect mode behave identically.

Client proxy mode

Best for: AWS SDK clients (boto3, AWS CLI, PyTorch S3 Connector) that need near-redirect-mode throughput. Use the Kubernetes Operator to deploy the S3 Proxy on nodes that run S3 clients, and set alluxio.worker.s3.redirect.enabled=true on the workers. Depending on the deployment mode, clients can access the local proxy through the Service created by the Operator (hostNetwork: false), or through http://127.0.0.1:8080 when the client Pod also uses the host network (hostNetwork: true). The proxy resolves the data owner from the consistent hash ring and forwards there in a single hop (loopback when that worker is co-located), avoiding cross-worker relay. This gives AWS SDK clients that cannot follow 307 redirects near-redirect-mode throughput.

┌───────────────────────────────────────────────┐
│  Client Node                                  │
│  ┌─────────────────┐   ┌───────────────────┐  │
│  │ Application Pod │ → │  S3 Proxy :8080   │  │
│  │ boto3 / AWS CLI │   │  hash ring lookup │  │
│  └─────────────────┘   └────────┬──────────┘  │
└───────────────────────────────┼───────────────┘
                                │ direct to data-owning worker
                                ▼
┌─────────────────────────────────────────────────┐
│  Alluxio Workers (port 29998)                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │ Worker 0 │  │ Worker 1 │  │ Worker N │       │
│  └──────────┘  └──────────┘  └──────────┘       │
└─────────────────────────────────────────────────┘

Prerequisites: set alluxio.worker.s3.api.enabled: "true" and alluxio.worker.s3.redirect.enabled: "true", and make sure the S3 Proxy can be scheduled on the target nodes. Port 8080 must be free on those nodes only when you use hostNetwork: true.

Operator deployment, key configuration, both Kubernetes access methods, file segmentation, and troubleshooting are covered in S3 Proxy.

Authentication

Alluxio's S3 API supports two authentication methods:

SIMPLE (Default) — Alluxio parses the AWS Signature V4 Authorization header to extract the username, but does not validate the signature.

Access Key: The Alluxio username to perform operations as. If omitted, operations run as the user that launched the worker process.
Secret Key: Any non-empty value. Required by the client to generate the signature, but ignored by Alluxio.

OIDC — For centralized identity management using OpenID Connect tokens, refer to the Authentication guide.

Performance

Key Alluxio Parameters

The following parameters control S3 API performance. Apply the recommended values for high-throughput workloads.

alluxio.worker.s3.redirect.enabled — Default: true (worker redirect mode); set to false for worker direct mode. When false, all cross-worker reads are proxied through the receiving worker, which adds an extra network hop and roughly halves small-object throughput. Keep it enabled only with clients that correctly follow HTTP 307 redirects to non-AWS endpoints (see Deployment Modes).
alluxio.worker.s3.connection.keep.alive.enabled — Default: false; Recommended: true. Reuses TCP connections across requests to reduce handshake overhead and improve throughput under high concurrency.
alluxio.worker.s3.connection.idle.max.time — Default: 0sec. Idle timeout for keep-alive connections; 0 means no timeout.
alluxio.worker.s3.access.logging.enabled — Default: false; Recommended: true for production. When false, only failed requests are logged; when true, every request is logged. Useful for auditing and debugging; disable during benchmarks to avoid I/O overhead.
alluxio.worker.s3.only.https.access — Default: false. When enabled, rejects non-HTTPS requests.
alluxio.worker.s3.redirect.health.check.enabled — Default: true; keep it enabled. Before issuing a 307 the worker checks the redirect target's liveness with a TTL-cached background TCP probe of its S3 port and skips a target whose latest probe failed, serving the request locally instead. Passive filtering through the worker availability monitor is always on and unaffected by this key. When disabled, redirects are issued optimistically, so clients can be redirected to a worker that is down. Version note: on releases before AI-3.8-15.1.15 this flag performed per-request health-check RPCs, which is why older tuning notes recommend disabling it for benchmarks; that per-request cost is gone, and the background probe (AI-3.8-15.1.18 and later) costs at most one TCP connect per target per TTL.
alluxio.worker.s3.redirect.liveness.ttl — Default: 1sec. How long a liveness verdict for a redirect target stays valid, whether the probe succeeded or failed; while a verdict is fresh no further probes are made for that target, and once it goes stale the next redirect triggers a background re-probe. It bounds both how long a just-died target keeps receiving redirects — so keep it below the S3 client's retry budget — and how long a just-restarted target keeps being served locally instead of redirected to. Only takes effect when alluxio.worker.s3.redirect.health.check.enabled is true.

Linux Kernel Parameters

For intensive S3 benchmarks with high TCP connection rates, tuning kernel parameters can improve connection reuse and lower latency.

sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_fin_timeout=30

# Only on older kernels (e.g., CentOS 7 / kernel 3.10):
sysctl -w net.ipv4.tcp_tw_recycle=1

Parameter

Effect

net.ipv4.tcp_tw_reuse

Reuses sockets in TIME_WAIT state, preventing port exhaustion during high-frequency requests.

net.ipv4.tcp_tw_recycle

Accelerates TIME_WAIT cleanup. Removed in Linux 4.12+ — this parameter only exists on older kernels (e.g., CentOS 7 with kernel 3.10). On newer kernels, this sysctl key does not exist and the command will fail.

net.ipv4.tcp_fin_timeout

Reduces idle connection close time, freeing resources faster. Default: 60s → recommended: 30s.

Warning: Modifying kernel parameters can impact system stability. Ensure you understand these settings before applying them, especially in production environments.
NAT Compatibility: On older kernels where tcp_tw_recycle is available, enabling it may cause connection failures for clients behind a NAT device due to timestamp validation issues. Do not use in NAT environments.

Advanced Features

MultiPartUpload (MPU)

For large object uploads that pass through to the underlying storage, these parameters control multipart upload behavior:

Parameter

Default

Recommended

Description

alluxio.underfs.object.store.multipart.upload.async

false

true

Enables async multipart uploads to UFS, improving write throughput.

alluxio.underfs.s3.upload.threads.max

20

256 for high-throughput

Maximum concurrent upload threads per worker for S3 UFS writes.

alluxio.underfs.s3.multipart.upload.buffer.number

64

256 for high-throughput

Number of multipart upload buffers. Increase alongside upload.threads.max.

By default, multipart uploads are pass-through — each part is uploaded directly to the underlying storage, and the object is committed there upon CompleteMultipartUpload. Alluxio does not buffer the parts locally.

If you need to write parts to Alluxio's cache layer first and asynchronously persist them to the underlying storage, see S3-API Write Optimization.

Tagging and Metadata

Enable Tagging: Requires extended attribute support for your UFS:
```
alluxio.underfs.xattr.change.enabled=true
```
Tag Limits: User-defined tags are limited to 10 per object/bucket per S3 tag restrictions. Disable with alluxio.worker.s3.tagging.restrictions.enabled=false.
Metadata Size: User-defined metadata is limited to 2KB per S3 metadata restrictions. Adjust with alluxio.worker.s3.header.metadata.max.size.

Limitations

Before adopting Alluxio's S3 API, be aware of these constraints:

Path-style only — Virtual-hosted style requests (http://<bucket>.<endpoint>/...) are not supported. All clients must use path-style (http://<endpoint>/<bucket>/<key>).
Buckets: Only top-level directories in the Alluxio namespace are treated as S3 buckets. The root directory (/) is not a bucket. To preserve existing S3 URIs, use the bucket name as the mount path:
```
alluxio mount add --path /<bucket-name> --ufs-uri s3://<bucket-name>/
```
No Versioning or Locking: If multiple clients write to the same object simultaneously, the last write wins.
Unsupported Characters: Object keys must not contain ?, \, ./, or ../; // may cause undefined behavior. For multipart uploads (MPU), object keys are additionally restricted to letters, digits, spaces, and the characters _ . : / = + - @ due to implementation constraints. Characters outside this set — such as &, #, %, !, ^, *, |, <, >, (, ), [, ], {, }, ", ', and ~ — are not supported in MPU object keys.
Folder Objects: Subdirectories are returned as 0-byte folder objects in ListObjects(V2) responses, matching AWS S3 console behavior.
Conditional reads: GetObject and HeadObject honor If-Match, If-None-Match, If-Modified-Since and If-Unmodified-Since from AI-3.8-15.1.20; earlier releases do not evaluate them. To keep a multi-request download on one version, see Which Version a Read Serves.
Signatures are not validated — Alluxio extracts the username from the Authorization header but does not verify the cryptographic signature. The secret key can be any non-empty string.
AWS SDK clients cannot use worker redirect mode — AWS SDK (AWS CLI), boto3, and PyTorch S3 Connector (AWS CRT) do not follow HTTP 307 redirects to non-AWS endpoints. Use worker direct mode (simplest) or client proxy mode (near-redirect throughput).
No virtual-hosted DNS — There is no DNS wildcard resolution for <bucket>.endpoint; this is a corollary of path-style only.

Supported S3 Actions

The following table lists supported S3 API actions. For detailed usage, see the official S3 API documentation.

S3 API Action

Supported Headers

Supported Query Parameters

AbortMultipartUpload

N/A

CompleteMultipartUpload

N/A

CopyObject

Content-Type, x-amz-copy-source, x-amz-metadata-directive, x-amz-tagging-directive, x-amz-tagging

N/A

CreateMultipartUpload

N/A

N/A

N/A

N/A

N/A

N/A

Range

N/A

N/A

N/A

N/A

N/A

N/A

N/A

delimiter, encoding-type, marker, max-keys, prefix

ListObjectsV2

N/A

continuation-token, delimiter, encoding-type, max-keys, prefix, start-after

ListParts

N/A

PutBucketTagging

N/A

PutObject

Content-Length, Content-MD5, Content-Type, x-amz-tagging

N/A

PutObjectTagging

N/A

UploadPart

Content-Length, Content-MD5

N/A

Usage Examples

minio-py — worker redirect mode

minio-py is a Python SDK that correctly follows HTTP 307 redirects to non-AWS endpoints. Use with a load balancer endpoint.

from minio import Minio

client = Minio(
    "<LOAD_BALANCER_ADDRESS>",   # host:port, no http:// prefix
    access_key="testuser",       # Alluxio username (or any value)
    secret_key="testpassword",   # Ignored by Alluxio
    secure=False,                # Set True if using HTTPS
)

# List buckets (Alluxio mount points)
for bucket in client.list_buckets():
    print(bucket.name)

# Download an object
response = client.get_object("<bucket>", "<object-key>")
data = response.read()
response.close()
response.release_conn()

# Upload an object
client.put_object("<bucket>", "<object-key>", data=open("file.txt", "rb"), length=-1, part_size=10*1024*1024)

Install with: pip install minio

boto3 — worker direct mode

boto3 does not follow HTTP 307 redirects to non-AWS endpoints. Use worker direct mode (or client proxy mode) with a load balancer endpoint.

import boto3

s3 = boto3.client(
    "s3",
    aws_access_key_id="placeholder",      # Alluxio username (or any value)
    aws_secret_access_key="placeholder",   # Ignored by Alluxio
    region_name="us-east-1",
    endpoint_url="http://<LOAD_BALANCER_ADDRESS>:29998",
)

# List buckets (Alluxio mount points)
response = s3.list_buckets()
for bucket in response.get("Buckets", []):
    print(f" - {bucket['Name']}")

Install with: pip install boto3

PyTorch S3 Connector — worker direct mode

PyTorch S3 Connector is built on AWS CRT and does not follow HTTP 307 redirects to non-AWS endpoints. Use worker direct mode (or client proxy mode) with a load balancer endpoint.

from s3torchconnector import S3IterableDataset, S3ClientConfig

s3_client_config = S3ClientConfig(force_path_style=True)

dataset = S3IterableDataset.from_prefix(
    "s3://s3-mount",                                    # Alluxio mount point
    region="us-east-1",
    endpoint="http://<LOAD_BALANCER_ADDRESS>:29998",
    s3client_config=s3_client_config,
)

for item in dataset:
    content = item.read()
    print(f"{item.key}: {len(content)} bytes")

Install with:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install s3torchconnector

Troubleshooting

AWS CLI, boto3, or PyTorch return errors or wrong data

Symptom: Operations fail with connection errors, or return unexpected results.

Cause: These clients are built on the AWS SDK (or AWS CRT), which does not follow HTTP 307 redirects to non-AWS endpoints. They require worker direct mode or client proxy mode — see Deployment Modes for setup instructions.

`NoSuchBucket` error even though the bucket exists in S3

Symptom: aws s3 ls s3://<bucket> or a GetObject returns NoSuchBucket.

Cause: In Alluxio's S3 API, buckets are Alluxio mount point names, not UFS bucket names. The bucket name in S3 requests must match the mount path in Alluxio (e.g., /s3 → bucket name s3), not the underlying S3 bucket name.

Diagnosis: List the actual bucket names Alluxio exposes:

AWS_ACCESS_KEY_ID=testuser AWS_SECRET_ACCESS_KEY=testpassword \
  aws s3 ls --endpoint-url http://<ENDPOINT>

Then use the name shown in the output — not the UFS bucket name — in subsequent requests.

Connection refused on port 29998

Symptom: aws s3 ls returns Could not connect to the endpoint URL.

Cause: The S3 API is not enabled on the workers, or the worker service is not reachable.

Fix:

Confirm S3 API is enabled:
```
kubectl -n alx-ns exec -i alluxio-cluster-worker-0 -- alluxio conf get alluxio.worker.s3.api.enabled
```
Expected: true. If not, add alluxio.worker.s3.api.enabled: "true" to spec.properties in alluxio-cluster.yaml and apply.

If running on Kubernetes without a load balancer, use port-forward:

kubectl -n alx-ns port-forward pod/alluxio-cluster-worker-0 29998:29998 &

`InvalidAccessKeyId` or signature errors

Cause: Alluxio's SIMPLE auth mode does not validate the signature, but the Authorization header must still be present and well-formed. Some clients omit it when no credentials are configured.

Fix: Pass any non-empty access key and secret key. With AWS CLI:

AWS_ACCESS_KEY_ID=testuser AWS_SECRET_ACCESS_KEY=testpassword \
  aws s3 ls --endpoint-url http://<ENDPOINT>