S3 Proxy

The S3 Proxy is a lightweight proxy service deployed on S3 client nodes. It uses the Alluxio mount table and consistent hash ring to identify the Alluxio worker that stores the requested object or file segment, then forwards the request directly to that worker.

The S3 Proxy is primarily intended for AWS CLI, boto3, PyTorch S3 Connector, and other AWS SDK-based clients that cannot correctly handle HTTP 307 redirects from Alluxio workers. Applications do not need to change how they call the S3 API; they only need to point their S3 endpoint to the S3 Proxy.

This page covers deploying and operating the proxy. For the S3 API itself — endpoints, authentication, deployment modes, and supported actions — see S3 API.

How It Works

S3 client (AWS CLI, boto3, PyTorch S3 Connector, and others)
    │  Path-style S3 request
    ▼
Node-local S3 Proxy (default port 8080)
    │  Selects a worker based on the mount table,
    │  consistent hash ring, and Range offset
    ▼
Alluxio Worker S3 API that owns the data (default port 29998)
    │  Cache hit: return data directly
    │  Cache miss: load data from UFS
    ▼
Underlying storage (S3, GCS, HDFS, and others)

After it starts, the S3 Proxy connects to the etcd instance used by the Alluxio cluster and retrieves the worker list, mount table, and routing information. The proxy refreshes this information periodically as workers join or leave the cluster.

For each request, the proxy:

1. Looks up the Alluxio mount table using the S3 bucket and object key.

2. Converts the S3 request path to the corresponding UFS path.

3. Calculates the worker that owns the data from the UFS path and consistent hash ring.

4. If file segmentation is enabled, calculates the worker that owns the segment using the starting offset of the Range request.

5. Forwards the request headers and body to the target worker's S3 API.

The proxy preserves request headers such as Authorization and Range. For several seconds immediately after startup, the proxy may briefly return 502 responses until it finishes synchronizing its initial worker and mount-table information.

Prerequisites

Before deploying the S3 Proxy, verify that:

• The current Kubernetes Operator and CRD support spec.s3Proxy. Support starts with Operator v3.7.2.

• The buckets that clients need to access through S3 are mounted in the Alluxio namespace.

• Kubernetes nodes can pull the S3 Proxy image.

• The S3 Proxy can be scheduled on every node that may run an S3 client.

• If you use hostNetwork: true, no other process on the node is using the default port 8080.

Check whether the Operator supports spec.s3Proxy:

kubectl explain alluxiocluster.spec.s3Proxy

Check the Alluxio cluster status. In this example, the namespace is alx-ns and the cluster name is alluxio-cluster:

kubectl -n alx-ns get alluxiocluster alluxio-cluster

Check the mount points:

kubectl -n alx-ns exec -it alluxio-cluster-coordinator-0 -- \
  alluxio mount list

The namespace, AlluxioCluster name, and Pod name above are examples. Replace them with the values from your environment.

Deploy with the Kubernetes Operator

In an environment managed by the Kubernetes Operator, you only need to update the existing AlluxioCluster resource. The Operator automatically creates and manages:

• A ConfigMap containing the S3 Proxy configuration

• A DaemonSet that runs one S3 Proxy Pod on each target node

• A ClusterIP Service for in-cluster access

• Container ports, health checks, and DNS policies

Do not create or edit the S3 Proxy ConfigMap, DaemonSet, or Service manually. The Operator may overwrite changes made directly to generated resources.

Configuration Example

The following example uses:

• alluxio-cluster as the AlluxioCluster name.

• alx-ns as the namespace.

• The default S3 Proxy port, 8080.

• hostNetwork: true, allowing clients to connect through the node loopback address 127.0.0.1.

• Alluxio-client: "true" to select nodes that run S3 clients.

• 1 GiB file segmentation. You can omit the segmentation settings if segmentation is not required.

Example configuration: Change the image location, image version, namespace, node labels, tolerations, and resource allocation for your environment. Port 8080 is the default, but you can change it through spec.s3Proxy.ports.http. If you change the port, clients must use the same port in their endpoint.

Merge the following fields into the existing alluxio-cluster.yaml. Do not overwrite or remove other existing settings under spec or spec.properties.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
metadata:
  name: alluxio-cluster
  namespace: alx-ns
spec:
  properties:
    # Required: enable the S3 API on every Alluxio worker.
    alluxio.worker.s3.api.enabled: "true"

    # Reference configuration for client proxy mode.
    # The S3 Proxy routes each request to the worker that owns the data;
    # cross-segment reads also require workers to perform redirects.
    alluxio.worker.s3.redirect.enabled: "true"

    # Recommended: reuse HTTP connections to the worker S3 API.
    alluxio.worker.s3.connection.keep.alive.enabled: "true"
    alluxio.worker.s3.redirect.health.check.enabled: "false"

    # This example enables 1 GiB file segmentation.
    alluxio.user.file.segment.enabled: "true"
    alluxio.user.file.segment.size: "1GB"

  s3Proxy:
    # The S3 Proxy is disabled by default and must be enabled explicitly.
    enabled: true

    # Example image location and version. Replace them with available values.
    image: image-registry/alluxio-s3-proxy
    imageTag: AI-3.8-15.1.0
    imagePullPolicy: IfNotPresent

    # true: the proxy uses the node network and is accessible at
    # 127.0.0.1:8080. The current Operator default is true.
    hostNetwork: true

    # Default S3 Proxy listen port. If you change it, update the client endpoint.
    ports:
      http: 8080

    # Deploy a proxy Pod on every node that may run an S3 client.
    # Replace this with the client-node label used in your environment.
    nodeSelector:
      Alluxio-client: "true"

    # Required only when client nodes have the corresponding NoSchedule taint.
    # A toleration permits scheduling onto a tainted node; it does not add a taint.
    tolerations:
      - key: "Alluxio-Client"
        operator: "Exists"
        effect: "NoSchedule"

    config:
      # Value is in bytes and must exactly match
      # alluxio.user.file.segment.size.
      segmentSize: 1073741824

    # These resource values are examples, not universal recommendations.
    # Adjust them for the worker count, concurrency, and object sizes.
    resources:
      limits:
        cpu: "16"
        memory: "4Gi"
      requests:
        cpu: "2"
        memory: "1Gi"

Apply the configuration:

kubectl apply -f alluxio-cluster.yaml

Key Configuration Settings

Setting	Required	Example	Description
alluxio.worker.s3.api.enabled	Yes	"true"	Enables the worker S3 API. By default, the S3 Proxy forwards requests to worker port 29998.
alluxio.worker.s3.redirect.enabled	Yes for this reference configuration	"true"	Allows workers to issue HTTP redirects and supports cross-segment reads. The S3 client connects only to the proxy and does not handle worker routing directly.
alluxio.worker.s3.connection.keep.alive.enabled	Recommended	"true"	Reuses HTTP connections and reduces connection overhead for highly concurrent requests.
alluxio.user.file.segment.enabled	Required with segmentation	"true"	Enables segmented file reads.
alluxio.user.file.segment.size	Required with segmentation	"1GB"	Worker segment size. It must match s3Proxy.config.segmentSize.
s3Proxy.enabled	Yes	true	Instructs the Operator to create the S3 Proxy resources. The default is false.
s3Proxy.image	Yes	Example image location	The S3 Proxy uses a dedicated image and does not inherit the Alluxio cluster image.
s3Proxy.imageTag	Yes	Example version	Use an image version that is compatible with and validated for the target Alluxio version.
s3Proxy.hostNetwork	Yes; set it explicitly	true	Determines whether clients use the node loopback address or Kubernetes Service. The current default is true.
s3Proxy.ports.http	No	8080	Default S3 Proxy port. The Operator applies it to the proxy listen port, container port, and Service port.
s3Proxy.nodeSelector	Environment-dependent	Example node label	Determines which nodes run the proxy. It must cover every node that may run an S3 client.
s3Proxy.tolerations	Environment-dependent	Example toleration	Required when client nodes have taints.
s3Proxy.config.segmentSize	Required with segmentation	1073741824	Proxy segment size in bytes. 1 GiB equals 1073741824 bytes.
s3Proxy.resources	Recommended	Example resources	Adjust for actual concurrency and cluster size; do not treat the example as a fixed recommendation.

The Operator derives the following proxy settings from the AlluxioCluster; you do not need to configure them manually:

• etcd endpoint

• Alluxio cluster name

• Worker S3 API port

• S3 Proxy listen port

• The ClusterFirstWithHostNet DNS policy when hostNetwork: true

• The ClusterFirst DNS policy when hostNetwork: false

Keys in nodeSelector and tolerations are case-sensitive. Alluxio-client and Alluxio-Client in this example are different keys, and each must match the actual node label or taint.

If the nodes do not yet have the example label, run:

kubectl label node <client-node-name> Alluxio-client=true

This command changes the node's labels. Before running it, verify that <client-node-name> is a node intended to run S3 clients.

Verify the Operator-Generated Resources

Use the component label to list all S3 Proxy resources created by the Operator:

kubectl -n alx-ns get daemonset,configmap,service \
  -l app.kubernetes.io/component=s3-proxy

With the AlluxioCluster name in this example, the Operator typically creates:

alluxio-cluster-s3-proxy
alluxio-cluster-s3-proxy-config
alluxio-cluster-s3-proxy-local

Actual resource names may vary based on the cluster name and Operator naming rules. Use the label query to discover the names instead of assuming fixed names in automation.

Wait for the DaemonSet to become ready:

kubectl -n alx-ns rollout status \
  daemonset/alluxio-cluster-s3-proxy

Check the nodes that run the proxy Pods:

kubectl -n alx-ns get pods \
  -l app.kubernetes.io/component=s3-proxy \
  -o wide

Check the DaemonSet status:

kubectl -n alx-ns get daemonset \
  -l app.kubernetes.io/component=s3-proxy

The DaemonSet DESIRED, CURRENT, and READY counts should match, and every node that may run an S3 client should have a Ready S3 Proxy Pod.

Access the S3 Proxy from Kubernetes Pods

Kubernetes clients can access the S3 Proxy in two ways:

Access method	S3 Proxy configuration	Client Pod configuration	Endpoint	Guaranteed to use the local-node proxy
Kubernetes Service	hostNetwork: false recommended	A regular Pod	http://<service-name>:8080	Yes; the Operator sets internalTrafficPolicy: Local
Node loopback	hostNetwork: true	The client must also use hostNetwork: true	http://127.0.0.1:8080	Yes

Current Operator behavior: When hostNetwork: true, the Operator still creates a ClusterIP Service, but does not set internalTrafficPolicy: Local on that Service. Requests through the Service work, but may be forwarded to an S3 Proxy on another node and are not guaranteed to remain node-local. To guarantee node-local access, use one of the two recommended configurations below.

Method 1: Access Through a Kubernetes Service

This method is intended for Kubernetes applications that use the regular Pod network and is recommended for most Kubernetes workloads.

Configure the S3 Proxy as follows:

spec:
  s3Proxy:
    hostNetwork: false
    ports:
      http: 8080

This fragment shows only the settings related to the access method. Keep the image, node selector, tolerations, segmentation, and resource settings from the complete example.

When s3Proxy.hostNetwork is false, the current Operator:

• Runs the S3 Proxy in the regular Pod network.

• Creates a ClusterIP Service.

• Automatically sets internalTrafficPolicy: Local on the Service.

• Does not configure a hostPort for the proxy.

• Automatically uses the ClusterFirst DNS policy.

internalTrafficPolicy: Local instructs kube-proxy to forward Service traffic only to a Ready endpoint on the same node as the client. This prevents the request from first crossing the network to an S3 Proxy on another node.

If the client node does not have a Ready S3 Proxy endpoint, the request fails instead of being forwarded to another node. Therefore, s3Proxy.nodeSelector must cover every S3 client node.

Verify the Service Configuration

Find the actual Service name:

kubectl -n alx-ns get service \
  -l app.kubernetes.io/component=s3-proxy

For the example cluster, the Service name is:

alluxio-cluster-s3-proxy-local

Verify the Service's local traffic policy:

kubectl -n alx-ns get service \
  alluxio-cluster-s3-proxy-local \
  -o jsonpath='{.spec.internalTrafficPolicy}{"\n"}'

Expected output:

Local

Check the Service endpoints and their nodes:

kubectl -n alx-ns get endpointslice \
  -l kubernetes.io/service-name=alluxio-cluster-s3-proxy-local \
  -o wide

If the client Pod's node has no corresponding endpoint, check the S3 Proxy nodeSelector, tolerations, and Pod readiness.

Service Endpoint

A client in the same namespace as the S3 Proxy can use the short Service name:

http://alluxio-cluster-s3-proxy-local:8080

A client in another namespace should use the fully qualified Service DNS name:

http://alluxio-cluster-s3-proxy-local.alx-ns.svc.cluster.local:8080

Check the health endpoint from an actual client Pod:

kubectl -n <client-namespace> exec <client-pod> -- \
  curl -fsS \
  http://alluxio-cluster-s3-proxy-local.alx-ns.svc.cluster.local:8080/health

Expected output:

OK

Do not use 127.0.0.1:8080 from a regular Kubernetes Pod. The loopback address of a regular Pod refers to the client Pod itself, not the node or the S3 Proxy Pod.

Access Through the Service with hostNetwork: true

The current Operator also creates a ClusterIP Service when s3Proxy.hostNetwork: true, but it does not set internalTrafficPolicy: Local. When the field is omitted, Kubernetes uses the Cluster policy and may select a Ready S3 Proxy endpoint on any node.

Therefore, this combination:

s3Proxy.hostNetwork: true + client uses Service DNS

is functional, but does not guarantee that requests use the local-node S3 Proxy and may add a cross-node network transfer. It is not recommended for workloads that depend on node-local performance.

Method 2: Access Through 127.0.0.1

Use this method when clients already use the host network or the application explicitly requires a node loopback endpoint.

Configure the S3 Proxy as follows:

spec:
  s3Proxy:
    hostNetwork: true
    ports:
      http: 8080

When s3Proxy.hostNetwork is true, the current Operator:

• Runs the S3 Proxy in the node network namespace.

• Configures ports.http as both the container port and hostPort.

• Automatically uses ClusterFirstWithHostNet, allowing the proxy to resolve in-cluster etcd and other Services.

• Creates a ClusterIP Service without setting internalTrafficPolicy: Local.

The client Pod must also use the host network. For example:

apiVersion: v1
kind: Pod
metadata:
  name: s3-client
  namespace: client-ns
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  nodeSelector:
    Alluxio-client: "true"
  tolerations:
    - key: "Alluxio-Client"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: client
      image: <client-image>
      env:
        - name: S3_ENDPOINT
          value: http://127.0.0.1:8080

This client Pod and a Ready S3 Proxy Pod must run on the same node. The S3 Proxy DaemonSet must cover every node where the client may be scheduled.

Check the health endpoint from the client Pod:

kubectl -n client-ns exec s3-client -- \
  curl -fsS http://127.0.0.1:8080/health

Expected output:

OK

Before using this method, verify that:

• No other process on the node uses port 8080.

• The client Pod sets hostNetwork: true.

• The client Pod and S3 Proxy Pod run on the same node.

• Network policies and node security policies permit the connection.

If the client uses the regular Pod network, 127.0.0.1 refers only to the client Pod, even when the client and S3 Proxy run on the same node.

Verify S3 API Access

The Alluxio S3 API supports only path-style requests. Virtual-hosted-style bucket addresses are not supported.

Set the endpoint for the selected access method:

# Method 1: access through a Kubernetes Service
export S3_PROXY_ENDPOINT=http://alluxio-cluster-s3-proxy-local.alx-ns.svc.cluster.local:8080

# Method 2: access through the node loopback address
# export S3_PROXY_ENDPOINT=http://127.0.0.1:8080

Verify with AWS CLI

Configure path-style addressing:

aws configure set default.s3.addressing_style path

With SIMPLE authentication, the Access Key is the Alluxio username and the Secret Key can be any non-empty value:

export AWS_ACCESS_KEY_ID=testuser
export AWS_SECRET_ACCESS_KEY=testpassword

List the buckets:

aws s3 ls --endpoint-url "${S3_PROXY_ENDPOINT}"

Upload and download a test object:

printf 's3-proxy-test\n' > /tmp/s3-proxy-test.txt

aws s3 cp /tmp/s3-proxy-test.txt \
  s3://<bucket>/s3-proxy-test.txt \
  --endpoint-url "${S3_PROXY_ENDPOINT}"

aws s3 cp \
  s3://<bucket>/s3-proxy-test.txt \
  /tmp/s3-proxy-download.txt \
  --endpoint-url "${S3_PROXY_ENDPOINT}"

cmp /tmp/s3-proxy-test.txt /tmp/s3-proxy-download.txt

If cmp produces no output and exits with status 0, the downloaded content matches the uploaded content.

Verify a Range Read with boto3

Install boto3 in the client environment and provide the endpoint and credentials through environment variables. Do not store production credentials directly in source code.

import os

import boto3
from botocore.config import Config


s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["S3_PROXY_ENDPOINT"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    region_name=os.environ.get("AWS_REGION", "us-east-1"),
    config=Config(s3={"addressing_style": "path"}),
)

response = s3.get_object(
    Bucket="<bucket>",
    Key="<object-key>",
    Range="bytes=0-1023",
)

try:
    data = response["Body"].read()
finally:
    response["Body"].close()

print(
    "status=",
    response["ResponseMetadata"]["HTTPStatusCode"],
    "content_range=",
    response.get("ContentRange"),
    "bytes=",
    len(data),
)

A successful Range request should return HTTP 206 and show the returned byte range in ContentRange.

Consistent Hashing and File Segmentation

The S3 Proxy must use the UFS path as the consistent-hash key instead of using the S3 request path directly. For example:

s3://real-bucket/data/file

The proxy uses the Alluxio mount table to convert the bucket and object key in a request to a UFS path. To ensure that the proxy and Alluxio workers calculate the same data ownership, correctly mount each bucket exposed through S3 in Alluxio:

alluxio mount add \
  --path /<bucket> \
  --ufs-uri s3://<real-bucket>/

If there is no matching mount, the proxy falls back to using s3://bucket/key for hashing. This result may differ from the worker's result for a mounted path, causing requests to be routed to the wrong worker.

When file segmentation is enabled, large files are divided into fixed-size segments that can be cached by different workers. The proxy uses the starting offset of a Range request to select the worker for the corresponding segment.

The following two values must match exactly:

spec:
  properties:
    alluxio.user.file.segment.size: "1GB"
  s3Proxy:
    config:
      segmentSize: 1073741824

s3Proxy.config.segmentSize is expressed in bytes. A mismatch can route Range requests to the wrong worker.

Update the S3 Proxy Configuration

Update spec.s3Proxy in the AlluxioCluster and apply the resource again:

kubectl apply -f alluxio-cluster.yaml

The Operator continues to manage the generated ConfigMap, DaemonSet, and Service. Do not edit generated resources directly.

The S3 Proxy mounts the generated config.yaml using subPath. If you change only fields under s3Proxy.config, existing proxy Pods do not reload the updated ConfigMap automatically. Restart the DaemonSet after the ConfigMap is updated:

kubectl -n alx-ns rollout restart \
  daemonset/alluxio-cluster-s3-proxy

Wait for the rollout to complete:

kubectl -n alx-ns rollout status \
  daemonset/alluxio-cluster-s3-proxy

Docker or Bare-Metal Deployment

In non-Kubernetes environments, the S3 Proxy uses a standalone structured config.yaml. Configure the etcd endpoint, Alluxio cluster name, worker S3 port, proxy listen port, and segment size manually.

Minimal configuration example:

etcd:
  urls: "http://alluxio-etcd:2379"
cluster:
  name: "DefaultAlluxioCluster"
worker:
  s3Port: 29998
proxy:
  listenPort: 8080
  segmentSize: 1073741824

Set ALLUXIO_PROXY_CONFIG to the configuration file and start the proxy:

export ALLUXIO_PROXY_CONFIG=./config.yaml
integration/proxy/start.sh

If etcd authentication is enabled, provide the credentials through environment variables:

export ALLUXIO_ETCD_USERNAME='<username>'
export ALLUXIO_ETCD_PASSWORD='<password>'

To run the container in the foreground, set:

export ALLUXIO_PROXY_FOREGROUND=1

Restart the S3 Proxy after changing config.yaml.

Troubleshooting

The DaemonSet Does Not Create Pods

Check the DaemonSet DESIRED count:

kubectl -n alx-ns get daemonset \
  -l app.kubernetes.io/component=s3-proxy

If DESIRED is 0, no nodes usually match s3Proxy.nodeSelector. Check the node labels:

kubectl get nodes --show-labels

The S3 Proxy Pod Remains Pending

Check the Pod events:

kubectl -n alx-ns describe pod <s3-proxy-pod>

In particular, check whether:

• Node taints have matching tolerations.

• Sufficient CPU and memory resources are available.

• nodeSelector matches the intended nodes.

Service DNS Resolves but the Connection Times Out

When internalTrafficPolicy: Local is active, kube-proxy does not forward traffic to another node if the client node has no Ready S3 Proxy endpoint.

Check the nodes that run the client and proxy Pods:

kubectl -n <client-namespace> get pod <client-pod> -o wide

kubectl -n alx-ns get pods \
  -l app.kubernetes.io/component=s3-proxy \
  -o wide

Then check the EndpointSlice:

kubectl -n alx-ns get endpointslice \
  -l kubernetes.io/service-name=alluxio-cluster-s3-proxy-local \
  -o wide

The Connection to 127.0.0.1:8080 Is Refused

Verify that:

• The S3 Proxy sets hostNetwork: true.

• The client Pod also sets hostNetwork: true.

• The client and S3 Proxy run on the same node.

• The client port matches s3Proxy.ports.http.

• No other process on the node uses the port.

The Health Check Succeeds but S3 Requests Fail

Verify that:

• The client uses path-style addressing.

• The target bucket is mounted in Alluxio.

• Both the Access Key and Secret Key are non-empty.

• The application's endpoint matches the selected access method.

• The proxy has retrieved the worker and mount-table information from etcd.

Requests Return 502

The proxy may briefly return 502 before loading its initial worker view. If 502 responses continue, check connectivity from the proxy to etcd, the cluster name, and worker status.

View the proxy logs:

kubectl -n alx-ns logs \
  -l app.kubernetes.io/component=s3-proxy \
  --all-containers=true \
  --tail=200

Range Requests Are Routed to the Wrong Worker

Verify that:

• alluxio.user.file.segment.enabled is true.

• alluxio.user.file.segment.size exactly matches s3Proxy.config.segmentSize.

• The bucket is mounted correctly, allowing the proxy to convert the S3 path to the correct UFS path.

• alluxio.worker.s3.redirect.enabled is true.

Compare the file locations with the target= field in the proxy log:

alluxio fs location <path> --segments 0-2

Requests for the same segment should be forwarded to the worker that stores that segment.