TLS Support

1. Introduction

TLS (Transport Layer Security) is a cryptographic protocol that ensures secure communication over the internet. This guide provides detailed instructions on configuring TLS support in Alluxio to secure RPCs and data transfers between various components.

Note: While ensuring data integrity and confidentiality during transmission, enabling TLS may also introduce performance overhead in data transmission.

Supported Certificate Formats

Alluxio supports two certificate formats for TLS configuration:

Java Keystore/Truststore (JKS): Traditional Java-based certificate storage
PEM Certificates: OpenSSL-compatible certificate format for modern deployments

2. TLS Architecture Overview

TLS Communication Scenarios

Client to S3 API: Secure client access through S3-compatible interface
Client to Gateway: Secure management API access
Coordinator-Worker Communication: Secure internal cluster communication
Client to Alluxio Servers: Direct client access to Alluxio services

3. Component-Level TLS Configuration

This section provides an overview of TLS configuration approaches for different Alluxio components. Each component can be configured independently based on your security requirements.

3.1 Configuration Overview

TLS configuration in Alluxio involves:

Certificate Format Selection: Choose between JKS or PEM certificates
Component-Specific Settings: Configure TLS for S3 API, Gateway, and inter-component communication
Security Level: Decide between TLS-only, dual-port (TLS + non-TLS), or mutual TLS (mTLS)

3.2 Prerequisites

Before configuring TLS, ensure you have:

Valid certificates (either self-signed or from a Certificate Authority)
Proper network connectivity between components
Sufficient storage and compute resources for TLS overhead

4. Certificate Configuration Details

This section covers certificate generation and management for both supported formats. Choose the format that best fits your environment and security requirements.

4.1 Using Java Keystore/Truststore (JKS)

The JKS format is the traditional Java-based approach for certificate storage, ideal for Java-centric environments.

Certificate Overview

Keystore: Stores private keys and certificates for entity identification during TLS handshaking
Truststore: Stores trusted certificates for verifying certificates received during TLS handshaking. e.g. Import the CA certificate into the truststore.

4.2 Using PEM Certificates

PEM certificates offer greater flexibility and are widely supported across different platforms and tools. This format is recommended for modern deployments, especially in containerized environments.

Generate CA and Server Certificates

Generate a Private Key for the Certificate Authority (CA):

openssl genrsa -out ca.key 2048

Generate a CA Certificate:

openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.pem \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=rootCA"

Generate a Private Key for the Server:

openssl genrsa -out server-key.pem 2048

Generate a Certificate Signing Request (CSR) for the Server:

For basic server certificate:

openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost"

For Alluxio worker in Kubernetes:

openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=serverhost" \
  -addext "subjectAltName = DNS:alluxio-worker.default.svc.cluster.local"

For Alluxio gateway in Kubernetes:

openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=serverhost" \
  -addext "subjectAltName = DNS:alluxio-gateway.default.svc.cluster.local"

Note: Add the Subject Alternative Name (SAN) parameter according to your environment requirements.

Sign the Server Certificate with the CA Key:

openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -out server.pem -days 365 -sha256

Generate Client Certificates

Generate a Private Key for the Client:

openssl genrsa -out client-key.pem 2048

Generate a Certificate Signing Request (CSR) for the Client:

openssl req -new -key client-key.pem -out client.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost"

Sign the Client Certificate with the CA Key:

openssl x509 -req -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -out client.pem -days 365 -sha256

5. Configuration Procedures

This section provides step-by-step configuration procedures for each Alluxio component. Follow the procedures relevant to your deployment architecture.

5.1 Alluxio S3 API Server TLS Configuration

The S3 API server supports flexible TLS configurations to meet different security and operational requirements.

Configuration Options

TLS-Only: Secure communication exclusively through HTTPS
Dual Port: Support both TLS and non-TLS connections simultaneously
Mutual TLS (mTLS): Enhanced security with client certificate verification

5.1.1 Enable TLS-Only for S3 API (Using JKS)

Add the following properties to alluxio-site.properties on the Alluxio Worker server:

# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=true

# Keystore configuration
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass

# Truststore configuration
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass

alluxio.network.tls.client.no.endpoint.identification=true
# Disable zero-copy for security
alluxio.worker.s3.local.page.transfer.enabled=false

5.1.2 Enable TLS-Only for S3 API (Using PEM)

# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=true

# PEM certificate configuration
alluxio.network.tls.ca.cert=<path to ca.pem>
alluxio.network.tls.server.cert=<path to server.pem>
alluxio.network.tls.server.key=<path to server-key.pem>
# Optional: Set private key passwords if your keys are encrypted
# alluxio.network.tls.server.key.password=your_server_key_password

# Disable zero-copy for security
alluxio.worker.s3.local.page.transfer.enabled=false

5.1.3 Enable Dual Port (TLS + Non-TLS) for S3 API

# Enable S3 endpoint
alluxio.worker.s3.api.enabled=true

# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.rest.port=29998
alluxio.worker.s3.only.https.access=false

# Certificate configuration (use either JKS or PEM format)
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass

# Performance settings
alluxio.worker.s3.local.page.transfer.enabled=false

5.1.4 Disable Non-TLS (TLS-Only)

To enforce HTTPS-only access, set:

alluxio.worker.s3.only.https.access=true
alluxio.worker.s3.https.port=29996

Important Notes:

Enabling HTTPS requires setting alluxio.worker.s3.local.page.transfer.enabled=false, which may impact performance
Both PEM and JKS certificate formats are supported
For mutual TLS (mTLS), add alluxio.worker.s3.tls.mutual.enabled=true
Default ports: HTTPS (29996), HTTP (29998)
Enabling TLS for the S3 API is distinct from enabling TLS between Alluxio Coordinaotr and Worker. If you just want to enable S3 TLS, you can set alluxio.network.tls.enabled=false

5.2 Coordinator and Worker TLS Configuration

Secure communication between Alluxio coordinators and workers is essential for cluster security. This configuration applies to all internal cluster communications. If you only need to enable TLS for the S3 API, you can skip this configuration section.

5.2.1 Using JKS Certificates

Add the following properties to alluxio-site.properties on all Alluxio servers:

# Enable TLS between servers
alluxio.network.tls.enabled=true

# Keystore configuration
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass

# Truststore configuration
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass

# Worker transfer type
alluxio.worker.network.netty.file.transfer=MAPPED

5.2.2 Using PEM Certificates

# Enable TLS between servers
alluxio.network.tls.enabled=true

# PEM certificate configuration
alluxio.network.tls.ca.cert=/alluxio/tls/ca.pem
alluxio.network.tls.server.cert=/alluxio/tls/server.pem
alluxio.network.tls.server.key=/alluxio/tls/server-key.pem

# Worker transfer type
alluxio.worker.network.netty.file.transfer=MAPPED

Note: Enabling TLS on workers requires setting alluxio.worker.network.netty.file.transfer to MAPPED, which may impact performance due to disabling Netty zero-copy optimization.

5.3 Alluxio Client Configuration

Configure Alluxio clients to establish secure connections with TLS-enabled servers. This configuration is essential for any application connecting to a TLS-enabled Alluxio cluster.

5.3.1 Using JKS Certificates

# Enable TLS for clients
alluxio.network.tls.enabled=true

# Truststore configuration for clients
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass
alluxio.network.tls.client.no.endpoint.identification=true

5.3.2 Using PEM Certificates

# Enable TLS for clients
alluxio.network.tls.enabled=true

# PEM certificate configuration
alluxio.network.tls.ca.cert=/alluxio/tls/ca.pem
alluxio.network.tls.client.no.endpoint.identification=true

5.4 Gateway TLS Configuration

The Alluxio Gateway provides management APIs and can be configured to support TLS connections. This is particularly important when the gateway is exposed to external networks.

Gateway Configuration Options

Dual Port Mode: Supports both TLS and non-TLS connections
TLS-Only Mode: Exclusively uses TLS for enhanced security

5.4.1 Enable Dual Port (TLS + Non-TLS) for Gateway

In Kubernetes environment, configure the gateway in alluxio-cluster.yaml:

spec:
  gateway:
    service:
      ports:
        api: 80          # Non-TLS port
        tlsApi: 8443     # TLS port
    tls:
      enabled: true
      secretName: "my-pem-secret"
      certFile: server.pem
      certKeyFile: server-key.pem
      caFile: ca.pem
      mtlsEnabled: true

5.4.2 Enable TLS-Only for Gateway

To disable non-TLS access and only support TLS:

spec:
  gateway:
    service:
      ports:
        api: 8443        # Only TLS port
    tls:
      enabled: true
      secretName: "my-pem-secret"
      certFile: server.pem
      certKeyFile: server-key.pem
      caFile: ca.pem
      mtlsEnabled: true

5.5 Advanced TLS Settings

These settings provide additional security hardening and compatibility options for specialized environments.

Protocol and Cipher Configuration

Configure TLS protocols and security parameters for enhanced protection:

Specify TLS Protocols: To restrict the server to certain TLS protocols:
```
alluxio.network.tls.server.protocols=TLSv1.2,TLSv1.3
```
Multiple Keys in Keystore: If multiple keys exist, specify the key alias:
```
alluxio.network.tls.keystore.alias=serverkey
```
Disable Client Hostname Verification: For flexible client access:
```
alluxio.network.tls.client.no.endpoint.identification=true
```

5.6 ETCD TLS Configuration

When using external ETCD for cluster coordination, TLS configuration ensures secure communication between Alluxio components and the ETCD cluster.

Prerequisites

External ETCD cluster with TLS enabled
Valid client certificates for ETCD authentication
PKCS8 format client keys (required for ETCD compatibility)

Configuration Steps

If your Alluxio cluster uses ETCD and requires TLS encryption, refer to the external etcd documentation for detailed setup instructions.

For Alluxio worker and coordinator to connect to ETCD with TLS:

alluxio.etcd.endpoints=https://<etcd-server>:2379
alluxio.etcd.tls.enabled=true
alluxio.etcd.tls.ca.cert=/path/to/etcd-certs/ca.crt
alluxio.etcd.tls.client.cert=/path/to/etcd-certs/client.crt
alluxio.etcd.tls.client.key=/path/to/etcd-certs/client-key.pem
alluxio.etcd.tls.client.key.password=<your_password>

Important Notes:

The client key must be in PKCS8 format for ETCD TLS connections

To convert a standard PEM key to PKCS8 format:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in client-key.pem -out client-key-pkcs8.pem

If using an encrypted PKCS8 key, provide the password via alluxio.etcd.tls.client.key.password

6. Kubernetes Deployment Configuration

Kubernetes deployments require special consideration for TLS configuration due to pod lifecycle management and secret handling.

6.1 Overview

Key considerations for Kubernetes TLS deployment:

Secret Management: Secure storage and distribution of certificates
Pod Configuration: Mounting certificates and configuring environment variables
Service Configuration: Proper port and service exposure settings

6.2 Prepare Kubernetes Secrets

Kubernetes secrets provide a secure way to store and distribute certificates to pods.

For JKS Certificates

Create Kubernetes secrets using the generated keystore and truststore:

kubectl -n alx-ns create secret generic alluxio-tls-keystore --from-file=/alluxio/keystore.jks
kubectl -n alx-ns create secret generic alluxio-tls-truststore --from-file=/alluxio/truststore.jks

For PEM Certificates

Create a Kubernetes secret with all necessary PEM files:

kubectl create secret generic my-pem-secret 
  --from-file=ca.pem=./ca.pem 
  --from-file=server.pem=./server.pem 
  --from-file=server-key.pem=./server-key.pem 
  --from-file=client.pem=./client.pem 
  --from-file=client-key.pem=./client-key.pem

6.3 Mount Secrets to Pods

Configure the secrets in the alluxio-cluster.yaml file under the spec.secrets section:

For JKS Certificates

spec:
  secrets:
    coordinator:
      alluxio-tls-keystore: /secrets/alluxio-tls-keystore
      alluxio-tls-truststore: /secrets/alluxio-tls-truststore
    worker:
      alluxio-tls-keystore: /secrets/alluxio-tls-keystore
      alluxio-tls-truststore: /secrets/alluxio-tls-truststore

For PEM Certificates

spec:
  secrets:
    coordinator:
      my-pem-secret: /alluxio/tls
    worker:
      my-pem-secret: /alluxio/tls

6.4 Configure Alluxio Properties in Kubernetes

Configure TLS properties in the Alluxio cluster specification. The following examples show configurations for S3 API TLS enablement:

For JKS Certificates

spec:
  properties:
  # Enable S3 endpoint on workers
    alluxio.worker.s3.api.enabled: "true"
    # enable s3 TLS
    alluxio.worker.s3.tls.enabled: "true"
    alluxio.worker.s3.tls.mutual.enabled: "true"
    alluxio.worker.s3.https.port: "29996"
    alluxio.worker.rest.port: "29998"
    alluxio.worker.s3.only.https.access: "false"
    alluxio.network.tls.server.protocols: TLSv1.2,TLSv1.3
    # Disable the zero-copy
    alluxio.worker.s3.local.page.transfer.enabled: "false"
    alluxio.worker.network.netty.file.transfer: "MAPPED"

    alluxio.network.tls.enabled: "true"
    alluxio.network.tls.keystore.path: /secrets/alluxio-tls-keystore/keystore.jks
    alluxio.network.tls.keystore.alias: serverkey
    alluxio.network.tls.keystore.password: keypass
    alluxio.network.tls.keystore.key.password: keypass
    alluxio.network.tls.truststore.path: /secrets/alluxio-tls-truststore/truststore.jks
    alluxio.network.tls.truststore.password: trustpass
    alluxio.network.tls.client.no.endpoint.identification: "true"

For PEM Certificates

spec:
  properties:
    # Enable S3 endpoint on workers
    alluxio.worker.s3.api.enabled: "true"
    # enable s3 TLS
    alluxio.worker.s3.tls.enabled: "true"
    alluxio.worker.s3.tls.mutual.enabled: "true"
    alluxio.worker.s3.https.port: "29996"
    alluxio.worker.rest.port: "29998"
    alluxio.worker.s3.only.https.access: "false"
    alluxio.network.tls.server.protocols: TLSv1.2,TLSv1.3

    alluxio.network.tls.ca.cert: "/alluxio/tls/ca.pem"
    alluxio.network.tls.server.cert: "/alluxio/tls/server.pem"
    alluxio.network.tls.server.key: "/alluxio/tls/server-key.pem"
    alluxio.network.tls.client.no.endpoint.identification: "true"
        # Disable the zero-copy
    alluxio.worker.s3.local.page.transfer.enabled: "false"
    alluxio.worker.network.netty.file.transfer: "MAPPED"

6.5 Deploy and Restart Services

Apply the configuration and restart services to enable TLS.

Deployment Steps

Apply the configuration:

kubectl apply -f alluxio-cluster.yaml

Restart worker deployment to apply new settings:

kubectl rollout restart deployment alluxio-worker

Verification

After deployment, verify that:

All pods are running successfully
TLS endpoints are accessible
Certificate mounting is working correctly

6.6 Configure TLS for Coordinator-Worker Communication

For enabling TLS between coordinators and workers, add the following to your alluxio-cluster.yaml:

spec:
  properties:
    alluxio.network.tls.enabled: "true"

7. Testing and Verification

Comprehensive testing ensures that TLS configuration is working correctly across all components. This section provides testing procedures for different scenarios.

7.1 Testing Overview

Test the following scenarios:

Connectivity: Verify TLS endpoints are accessible
Certificate Validation: Ensure proper certificate chain verification
Security: Confirm non-TLS connections are properly blocked when required

7.2 Verify S3 API TLS Configuration

Test S3 non-TLS (when dual port is enabled):

curl http://<alluxio_worker_svc>:29998/test -v

Test S3 TLS:

curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
  https://<alluxio_worker_svc>:29996/test -v

7.3 Verify Gateway TLS Configuration

Test Gateway API endpoints to ensure proper TLS functionality.

Test Gateway non-TLS (when dual port is enabled):

curl -sS 'http://<alluxio_gateway_svc>:80/api/v1/mount' -v

Test Gateway TLS:

curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
  https://<alluxio_gateway_svc>:8443/api/v1/mount' -v

7.4 Verify TLS-Only Configuration

When TLS-only mode is enabled, verify that non-TLS connections are properly rejected:

S3 non-TLS (should fail with connection refused):

curl http://alluxio-worker.default.svc.cluster.local:29998/test -v

Expected output:

* connect to 10.0.5.157 port 29998 failed: Connection refused
* Failed to connect to alluxio-worker.default.svc.cluster.local port 29998: Connection refused
curl: (7) Failed to connect to alluxio-worker.default.svc.cluster.local port 29998: Connection refused

Gateway non-TLS (should fail with connection timeout):

curl -sS 'http://alluxio-gateway.default.svc.cluster.local:80/api/v1/mount' -v

Expected output:

* connect to 172.20.66.21 port 80 failed: Connection timed out
* Failed to connect to alluxio-gateway.default.svc.cluster.local port 80: Connection timed out
curl: (28) Failed to connect to alluxio-gateway.default.svc.cluster.local port 80: Connection timed out

Note: Replace test with your actual mount point name in the S3 tests.

8. Security Best Practices

Implementing TLS correctly requires attention to certificate management, operational security, and ongoing maintenance. This section outlines industry best practices for TLS deployment.

8.1 Certificate Lifecycle Management

Proper certificate management is crucial for maintaining secure operations.

Certificate Management and Renewal

Regular Certificate Rotation: Implement automated certificate renewal processes
Secure Storage: Store private keys and certificates in secure locations with appropriate file permissions
Backup Strategy: Maintain secure backups of certificates and keys

Key and Certificate Protection

Implement proper security measures for certificate storage and access:

File Permissions: Ensure certificate files are readable only by the Alluxio processes
```
chmod 600 /alluxio/tls/*.key
chmod 644 /alluxio/tls/*.pem
```
Access Control: Implement proper access controls for certificate directories
Monitoring: Set up monitoring for certificate expiration dates

Multi-Environment Deployment Recommendations

Implement proper isolation and management practices across environments:

Environment Separation: Use different certificates for different environments (dev, staging, production)
Namespace Isolation: In Kubernetes, use different namespaces and secrets for different environments
Configuration Management: Use configuration management tools to maintain consistent TLS settings

9. Appendix

This appendix provides additional resources, command references, and troubleshooting guidance for TLS configuration and management.

9.1 Certificate Format Conversion

Convert between different certificate formats as needed for your environment.

Convert PEM to JKS

# Convert PEM certificate to PKCS12
openssl pkcs12 -export -in server.pem -inkey server-key.pem -out server.p12 -name alluxio

# Convert PKCS12 to JKS
keytool -importkeystore -deststorepass keypass -destkeystore keystore.jks \
  -srckeystore server.p12 -srcstoretype PKCS12 -srcstorepass keypass

Convert JKS to PEM

# Export certificate from JKS to PEM
keytool -exportcert -keystore keystore.jks -alias key -rfc -file server.pem

# Export private key (requires conversion through PKCS12)
keytool -importkeystore -srckeystore keystore.jks -destkeystore server.p12 \
  -deststoretype PKCS12 -srcalias key -deststorepass keypass -destkeypass keypass

openssl pkcs12 -in server.p12 -nocerts -out server-key.pem -nodes

9.2 Common Commands Reference

Essential commands for certificate management and TLS troubleshooting.

Certificate Information

# View PEM certificate details
openssl x509 -in server.pem -text -noout

# View JKS keystore contents
keytool -list -v -keystore keystore.jks

# Verify certificate chain
openssl verify -CAfile ca.pem server.pem

# Show certificate chain details
openssl crl2pkcs7 -nocrl -certfile ca.pem | openssl pkcs7 -print_certs -noout
openssl crl2pkcs7 -nocrl -certfile server.pem | openssl pkcs7 -print_certs -noout

# Check certificate expiration
openssl x509 -in server.pem -noout -dates

# Validate certificate against CA
openssl x509 -in server.pem -noout -issuer
openssl x509 -in ca.pem -noout -subject

Testing TLS Connectivity

# Test TLS connection
openssl s_client -connect hostname:port -cert client.pem -key client-key.pem -CAfile ca.pem

# Test without client certificate
openssl s_client -connect hostname:port -CAfile ca.pem

# Test with tls protocol (-tls1_2/ -tls1_3)
openssl s_client -connect localhost:29998 -cert client.pem -key client-key.pem -CAfile client-chain.pem -tls1_3

9.3 Troubleshooting Common Issues

Solutions for frequently encountered TLS configuration and connectivity problems.

Certificate Validation Errors

Verify certificate chain and CA certificates
Check certificate expiration dates
Ensure proper file permissions

Connection Failures

Verify port configurations
Check firewall settings
Validate DNS resolution for hostnames

Performance Issues

Monitor the impact of disabling zero-copy transfers
Consider certificate size and algorithms
Evaluate TLS protocol versions

Last updated 10 days ago