Securing Alluxio with TLS

Transport Layer Security (TLS) is a cryptographic protocol that provides secure communication for Alluxio services. By enabling TLS, you can ensure that data in transit between clients and Alluxio, as well as between Alluxio components themselves, is encrypted and protected from eavesdropping.

This guide provides a comprehensive walkthrough for configuring TLS in Alluxio, from generating certificates to deploying and verifying a secure cluster.

Note: Enabling TLS introduces computational overhead, which may impact data transfer performance.

Certificate Formats

Alluxio supports two standard certificate formats:

• PEM: The modern, recommended format, compatible with OpenSSL and widely used in containerized environments.

• Java Keystore (JKS): A traditional Java-native format.

This guide focuses on the PEM format. For instructions on converting between formats, see the Appendix.

Part 1: Generate Certificates

Before enabling TLS, you must have a set of certificates to authenticate your services. The following steps use openssl to create a simple Certificate Authority (CA) and issue certificates for servers and clients.

1. Create a Certificate Authority (CA)

The CA is used to sign and validate all the certificates in your cluster.

# Generate a private key for the CA
openssl genrsa -out ca.key 2048

# Generate a self-signed CA certificate
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.pem \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=rootCA"

2. Generate a Server Certificate

Each Alluxio server (Coordinator, Worker, Gateway) needs a server certificate.

# Generate a private key for the server
openssl genrsa -out server-key.pem 2048

# Generate a Certificate Signing Request (CSR) for the server
# Note: The CN and subjectAltName (SAN) must match the hostname clients use to connect.
# Example for a worker in Kubernetes:
openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=serverhost" \
  -addext "subjectAltName = DNS:alluxio-worker.default.svc.cluster.local"

# Sign the server certificate with your CA
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -out server.pem -days 365 -sha256

3. Generate a Client Certificate

Client certificates are required for mutual TLS (mTLS), where the server also verifies the client's identity.

# Generate a private key for the client
openssl genrsa -out client-key.pem 2048

# Generate a Certificate Signing Request (CSR) for the client
openssl req -new -key client-key.pem -out client.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost"

# Sign the client certificate with your CA
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -out client.pem -days 365 -sha256

Part 2: Configure Alluxio Services

With the certificates ready, you can now configure Alluxio services to use them. The following properties should be set in alluxio-site.properties.

Securing Internal Cluster Communication

This configuration encrypts RPC traffic between Alluxio Coordinators and Workers.

# Enable TLS between servers
alluxio.network.tls.enabled=true

# PEM certificate configuration
alluxio.network.tls.ca.cert=/path/to/ca.pem
alluxio.network.tls.server.cert=/path/to/server.pem
alluxio.network.tls.server.key=/path/to/server-key.pem

# Note: Enabling TLS requires setting the transfer type to MAPPED, which disables zero-copy.
alluxio.worker.network.netty.file.transfer=MAPPED

Securing the S3 API

You can secure the Alluxio S3 API endpoint on the workers. This is configured independently from the internal RPC encryption.

Option 1: TLS-Only Mode

This mode enforces HTTPS for all S3 API access.

# Enable TLS for the S3 API
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=true

# PEM certificate configuration
alluxio.network.tls.ca.cert=/path/to/ca.pem
alluxio.network.tls.server.cert=/path/to/server.pem
alluxio.network.tls.server.key=/path/to/server-key.pem

# Disable zero-copy for security
alluxio.worker.s3.local.page.transfer.enabled=false

Option 2: Dual-Port Mode (HTTP and HTTPS)

This mode allows both secure (HTTPS) and insecure (HTTP) connections simultaneously.

# Enable TLS for the S3 API but allow non-TLS access
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=false
# Specify the non-TLS port
alluxio.worker.rest.port=29998

# (Add PEM certificate configuration as shown in Option 1)

Option 3: Mutual TLS (mTLS)

For maximum security, enable mTLS to require clients to present a valid certificate.

# Add this property to your existing S3 TLS configuration
alluxio.worker.s3.tls.mutual.enabled=true

Securing the Gateway

The Alluxio Gateway provides management APIs and can be configured to support TLS connections. This is particularly important when the gateway is exposed to external networks.

Note: Gateway TLS settings are typically configured in the alluxio-cluster.yaml for Kubernetes deployments.

Option 1: TLS-Only Mode

This mode enforces HTTPS for all Gateway API access.

# In alluxio-cluster.yaml
spec:
  gateway:
    tls:
      enabled: true
      secretName: "my-pem-secret"
      certFile: server.pem
      certKeyFile: server-key.pem
      caFile: ca.pem
      mtlsEnabled: true
    service:
      ports:
        api: 8443 # Only the TLS port is used

Option 2: Dual-Port Mode (HTTP and HTTPS)

This mode allows both secure (HTTPS) and insecure (HTTP) connections simultaneously.

# In alluxio-cluster.yaml
spec:
  gateway:
    tls:
      enabled: true
      secretName: "my-pem-secret"
      certFile: server.pem
      certKeyFile: server-key.pem
      caFile: ca.pem
      mtlsEnabled: true
    service:
      ports:
        api: 80      # Non-TLS port
        tlsApi: 8443 # TLS port

Configuring the Alluxio Client

Clients connecting to a TLS-enabled Alluxio cluster must be configured to trust the server's CA.

# Enable TLS for clients
alluxio.network.tls.enabled=true

# Provide the CA certificate for the client to trust
alluxio.network.tls.ca.cert=/path/to/ca.pem

# Optional: Disable hostname verification if needed (e.g., for testing)
alluxio.network.tls.client.no.endpoint.identification=true

Securing ETCD Communication

If you use an external ETCD cluster for service discovery, you can secure the connection to it.

alluxio.etcd.endpoints=https://<etcd-server>:2379
alluxio.etcd.tls.enabled=true
alluxio.etcd.tls.ca.cert=/path/to/etcd-ca.crt
alluxio.etcd.tls.client.cert=/path/to/etcd-client.crt
alluxio.etcd.tls.client.key=/path/to/etcd-client-key-pkcs8.pem
# alluxio.etcd.tls.client.key.password=<your_password>

Important: The ETCD client key must be in PKCS8 format. Convert a standard PEM key with:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in client-key.pem -out client-key-pkcs8.pem

Advanced: Specifying TLS Protocols

For enhanced security, you can restrict the server to specific TLS protocol versions. Add the following property to alluxio-site.properties:

# Specify allowed TLS protocols
alluxio.network.tls.server.protocols=TLSv1.2,TLSv1.3

Part 3: Deploying with TLS in Kubernetes

Configuring TLS in Kubernetes involves using secrets to manage certificates and applying the configuration via the Helm chart or alluxio-cluster.yaml.

1. Create a Kubernetes Secret

Store your generated PEM certificates in a Kubernetes secret.

kubectl create secret generic my-pem-secret \
  --from-file=ca.pem=./ca.pem \
  --from-file=server.pem=./server.pem \
  --from-file=server-key.pem=./server-key.pem \
  --from-file=client.pem=./client.pem \
  --from-file=client-key.pem=./client-key.pem

2. Configure and Mount the Secret

In your alluxio-cluster.yaml, reference the secret and configure the TLS properties.

spec:
  # Mount the secret to the pods at the specified path
  secrets:
    coordinator:
      my-pem-secret: /opt/alluxio/certs
    worker:
      my-pem-secret: /opt/alluxio/certs

  properties:
    # Enable TLS for S3 API (example)
    alluxio.worker.s3.api.enabled: "true"
    alluxio.worker.s3.tls.enabled: "true"
    alluxio.worker.s3.https.port: "29996"
    alluxio.worker.s3.only.https.access: "true"

    # Point to the certificate files mounted from the secret
    alluxio.network.tls.ca.cert: "/opt/alluxio/certs/ca.pem"
    alluxio.network.tls.server.cert: "/opt/alluxio/certs/server.pem"
    alluxio.network.tls.server.key: "/opt/alluxio/certs/server-key.pem"

    # Disable zero-copy for security
    alluxio.worker.s3.local.page.transfer.enabled: "false"

3. Deploy the Cluster

Apply the configuration to your Kubernetes cluster.

kubectl apply -f alluxio-cluster.yaml

Part 4: Verify the TLS Setup

After deploying, use tools like curl or openssl to verify that the TLS configuration is working correctly.

Verify S3 API (mTLS)

curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
  https://<alluxio_worker_svc>:29996/test -v

Verify S3 API (TLS-Only Mode)

When alluxio.worker.s3.only.https.access is true, attempting to connect via HTTP should fail.

curl http://<alluxio_worker_svc>:29998/test -v

This command should result in a "Connection refused" error.

Verify Gateway (mTLS)

curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
  https://<alluxio_gateway_svc>:8443/api/v1/mount -v

Part 5: Security Best Practices

• Certificate Rotation: Regularly renew and rotate your certificates to limit the window of exposure if a key is compromised.

• Secure Key Storage: Protect your private keys. On the filesystem, use strict permissions:

  chmod 600 /path/to/keys/*.key
  chmod 644 /path/to/certs/*.pem

• Environment Isolation: Use separate CAs and certificates for development, staging, and production environments.

Appendix

PEM to JKS Format Conversion

# 1. Convert PEM certificate to PKCS12
openssl pkcs12 -export -in server.pem -inkey server-key.pem -out server.p12 -name alluxio

# 2. Convert PKCS12 to JKS
keytool -importkeystore -deststorepass keypass -destkeystore keystore.jks \
  -srckeystore server.p12 -srcstoretype PKCS12 -srcstorepass keypass