TLS Support
1. Introduction
TLS (Transport Layer Security) is a cryptographic protocol that ensures secure communication over the internet. This guide provides detailed instructions on configuring TLS support in Alluxio to secure RPCs and data transfers between various components.
Note: While ensuring data integrity and confidentiality during transmission, enabling TLS may also introduce performance overhead in data transmission.
Supported Certificate Formats
Alluxio supports two certificate formats for TLS configuration:
Java Keystore/Truststore (JKS): Traditional Java-based certificate storage
PEM Certificates: OpenSSL-compatible certificate format for modern deployments
2. TLS Architecture Overview
TLS Communication Scenarios
Client to S3 API: Secure client access through S3-compatible interface
Client to Gateway: Secure management API access
Coordinator-Worker Communication: Secure internal cluster communication
Client to Alluxio Servers: Direct client access to Alluxio services
3. Component-Level TLS Configuration
This section provides an overview of TLS configuration approaches for different Alluxio components. Each component can be configured independently based on your security requirements.
3.1 Configuration Overview
TLS configuration in Alluxio involves:
Certificate Format Selection: Choose between JKS or PEM certificates
Component-Specific Settings: Configure TLS for S3 API, Gateway, and inter-component communication
Security Level: Decide between TLS-only, dual-port (TLS + non-TLS), or mutual TLS (mTLS)
3.2 Prerequisites
Before configuring TLS, ensure you have:
Valid certificates (either self-signed or from a Certificate Authority)
Proper network connectivity between components
Sufficient storage and compute resources for TLS overhead
4. Certificate Configuration Details
This section covers certificate generation and management for both supported formats. Choose the format that best fits your environment and security requirements.
4.1 Using Java Keystore/Truststore (JKS)
The JKS format is the traditional Java-based approach for certificate storage, ideal for Java-centric environments.
Certificate Overview
Keystore: Stores private keys and certificates for entity identification during TLS handshaking
Truststore: Stores trusted certificates for verifying certificates received during TLS handshaking. e.g. Import the CA certificate into the truststore.
4.2 Using PEM Certificates
PEM certificates offer greater flexibility and are widely supported across different platforms and tools. This format is recommended for modern deployments, especially in containerized environments.
Generate CA and Server Certificates
Generate a Private Key for the Certificate Authority (CA):
openssl genrsa -out ca.key 2048
Generate a CA Certificate:
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.pem \
-subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=rootCA"
Generate a Private Key for the Server:
openssl genrsa -out server-key.pem 2048
Generate a Certificate Signing Request (CSR) for the Server:
For basic server certificate:
openssl req -new -key server-key.pem -out server.csr \
-subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost"
For Alluxio worker in Kubernetes:
openssl req -new -key server-key.pem -out server.csr \
-subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=serverhost" \
-addext "subjectAltName = DNS:alluxio-worker.default.svc.cluster.local"
For Alluxio gateway in Kubernetes:
openssl req -new -key server-key.pem -out server.csr \
-subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=serverhost" \
-addext "subjectAltName = DNS:alluxio-gateway.default.svc.cluster.local"
Note: Add the Subject Alternative Name (SAN) parameter according to your environment requirements.
Sign the Server Certificate with the CA Key:
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
-out server.pem -days 365 -sha256
Generate Client Certificates
Generate a Private Key for the Client:
openssl genrsa -out client-key.pem 2048
Generate a Certificate Signing Request (CSR) for the Client:
openssl req -new -key client-key.pem -out client.csr \
-subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost"
Sign the Client Certificate with the CA Key:
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
-out client.pem -days 365 -sha256
5. Configuration Procedures
This section provides step-by-step configuration procedures for each Alluxio component. Follow the procedures relevant to your deployment architecture.
5.1 Alluxio S3 API Server TLS Configuration
The S3 API server supports flexible TLS configurations to meet different security and operational requirements.
Configuration Options
TLS-Only: Secure communication exclusively through HTTPS
Dual Port: Support both TLS and non-TLS connections simultaneously
Mutual TLS (mTLS): Enhanced security with client certificate verification
5.1.1 Enable TLS-Only for S3 API (Using JKS)
Add the following properties to alluxio-site.properties
on the Alluxio Worker server:
# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=true
# Keystore configuration
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass
# Truststore configuration
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass
alluxio.network.tls.client.no.endpoint.identification=true
# Disable zero-copy for security
alluxio.worker.s3.local.page.transfer.enabled=false
5.1.2 Enable TLS-Only for S3 API (Using PEM)
# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.s3.only.https.access=true
# PEM certificate configuration
alluxio.network.tls.ca.cert=<path to ca.pem>
alluxio.network.tls.server.cert=<path to server.pem>
alluxio.network.tls.server.key=<path to server-key.pem>
# Optional: Set private key passwords if your keys are encrypted
# alluxio.network.tls.server.key.password=your_server_key_password
# Disable zero-copy for security
alluxio.worker.s3.local.page.transfer.enabled=false
5.1.3 Enable Dual Port (TLS + Non-TLS) for S3 API
# Enable S3 endpoint
alluxio.worker.s3.api.enabled=true
# Enable TLS
alluxio.worker.s3.tls.enabled=true
alluxio.worker.s3.https.port=29996
alluxio.worker.rest.port=29998
alluxio.worker.s3.only.https.access=false
# Certificate configuration (use either JKS or PEM format)
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass
# Performance settings
alluxio.worker.s3.local.page.transfer.enabled=false
5.1.4 Disable Non-TLS (TLS-Only)
To enforce HTTPS-only access, set:
alluxio.worker.s3.only.https.access=true
alluxio.worker.s3.https.port=29996
Important Notes:
Enabling HTTPS requires setting
alluxio.worker.s3.local.page.transfer.enabled=false
, which may impact performanceBoth PEM and JKS certificate formats are supported
For mutual TLS (mTLS), add
alluxio.worker.s3.tls.mutual.enabled=true
Default ports: HTTPS (29996), HTTP (29998)
Enabling TLS for the S3 API is distinct from enabling TLS between Alluxio Coordinaotr and Worker. If you just want to enable S3 TLS, you can set
alluxio.network.tls.enabled=false
5.2 Coordinator and Worker TLS Configuration
Secure communication between Alluxio coordinators and workers is essential for cluster security. This configuration applies to all internal cluster communications. If you only need to enable TLS for the S3 API, you can skip this configuration section.
5.2.1 Using JKS Certificates
Add the following properties to alluxio-site.properties
on all Alluxio servers:
# Enable TLS between servers
alluxio.network.tls.enabled=true
# Keystore configuration
alluxio.network.tls.keystore.path=/alluxio/keystore.jks
alluxio.network.tls.keystore.password=keypass
alluxio.network.tls.keystore.alias=serverkey
alluxio.network.tls.keystore.key.password=keypass
# Truststore configuration
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass
# Worker transfer type
alluxio.worker.network.netty.file.transfer=MAPPED
5.2.2 Using PEM Certificates
# Enable TLS between servers
alluxio.network.tls.enabled=true
# PEM certificate configuration
alluxio.network.tls.ca.cert=/alluxio/tls/ca.pem
alluxio.network.tls.server.cert=/alluxio/tls/server.pem
alluxio.network.tls.server.key=/alluxio/tls/server-key.pem
# Worker transfer type
alluxio.worker.network.netty.file.transfer=MAPPED
Note: Enabling TLS on workers requires setting alluxio.worker.network.netty.file.transfer
to MAPPED
, which may impact performance due to disabling Netty zero-copy optimization.
5.3 Alluxio Client Configuration
Configure Alluxio clients to establish secure connections with TLS-enabled servers. This configuration is essential for any application connecting to a TLS-enabled Alluxio cluster.
5.3.1 Using JKS Certificates
# Enable TLS for clients
alluxio.network.tls.enabled=true
# Truststore configuration for clients
alluxio.network.tls.truststore.path=/alluxio/truststore.jks
alluxio.network.tls.truststore.password=trustpass
alluxio.network.tls.client.no.endpoint.identification=true
5.3.2 Using PEM Certificates
# Enable TLS for clients
alluxio.network.tls.enabled=true
# PEM certificate configuration
alluxio.network.tls.ca.cert=/alluxio/tls/ca.pem
alluxio.network.tls.client.no.endpoint.identification=true
5.4 Gateway TLS Configuration
The Alluxio Gateway provides management APIs and can be configured to support TLS connections. This is particularly important when the gateway is exposed to external networks.
Gateway Configuration Options
Dual Port Mode: Supports both TLS and non-TLS connections
TLS-Only Mode: Exclusively uses TLS for enhanced security
5.4.1 Enable Dual Port (TLS + Non-TLS) for Gateway
In Kubernetes environment, configure the gateway in alluxio-cluster.yaml
:
spec:
gateway:
service:
ports:
api: 80 # Non-TLS port
tlsApi: 8443 # TLS port
tls:
enabled: true
secretName: "my-pem-secret"
certFile: server.pem
certKeyFile: server-key.pem
caFile: ca.pem
mtlsEnabled: true
5.4.2 Enable TLS-Only for Gateway
To disable non-TLS access and only support TLS:
spec:
gateway:
service:
ports:
api: 8443 # Only TLS port
tls:
enabled: true
secretName: "my-pem-secret"
certFile: server.pem
certKeyFile: server-key.pem
caFile: ca.pem
mtlsEnabled: true
5.5 Advanced TLS Settings
These settings provide additional security hardening and compatibility options for specialized environments.
Protocol and Cipher Configuration
Configure TLS protocols and security parameters for enhanced protection:
Specify TLS Protocols: To restrict the server to certain TLS protocols:
alluxio.network.tls.server.protocols=TLSv1.2,TLSv1.3
Multiple Keys in Keystore: If multiple keys exist, specify the key alias:
alluxio.network.tls.keystore.alias=serverkey
Disable Client Hostname Verification: For flexible client access:
alluxio.network.tls.client.no.endpoint.identification=true
5.6 ETCD TLS Configuration
When using external ETCD for cluster coordination, TLS configuration ensures secure communication between Alluxio components and the ETCD cluster.
Prerequisites
External ETCD cluster with TLS enabled
Valid client certificates for ETCD authentication
PKCS8 format client keys (required for ETCD compatibility)
Configuration Steps
If your Alluxio cluster uses ETCD and requires TLS encryption, refer to the external etcd documentation for detailed setup instructions.
For Alluxio worker and coordinator to connect to ETCD with TLS:
alluxio.etcd.endpoints=https://<etcd-server>:2379
alluxio.etcd.tls.enabled=true
alluxio.etcd.tls.ca.cert=/path/to/etcd-certs/ca.crt
alluxio.etcd.tls.client.cert=/path/to/etcd-certs/client.crt
alluxio.etcd.tls.client.key=/path/to/etcd-certs/client-key.pem
alluxio.etcd.tls.client.key.password=<your_password>
Important Notes:
The client key must be in PKCS8 format for ETCD TLS connections
To convert a standard PEM key to PKCS8 format:
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in client-key.pem -out client-key-pkcs8.pem
If using an encrypted PKCS8 key, provide the password via
alluxio.etcd.tls.client.key.password
6. Kubernetes Deployment Configuration
Kubernetes deployments require special consideration for TLS configuration due to pod lifecycle management and secret handling.
6.1 Overview
Key considerations for Kubernetes TLS deployment:
Secret Management: Secure storage and distribution of certificates
Pod Configuration: Mounting certificates and configuring environment variables
Service Configuration: Proper port and service exposure settings
6.2 Prepare Kubernetes Secrets
Kubernetes secrets provide a secure way to store and distribute certificates to pods.
For JKS Certificates
Create Kubernetes secrets using the generated keystore and truststore:
kubectl -n alx-ns create secret generic alluxio-tls-keystore --from-file=/alluxio/keystore.jks
kubectl -n alx-ns create secret generic alluxio-tls-truststore --from-file=/alluxio/truststore.jks
For PEM Certificates
Create a Kubernetes secret with all necessary PEM files:
kubectl create secret generic my-pem-secret
--from-file=ca.pem=./ca.pem
--from-file=server.pem=./server.pem
--from-file=server-key.pem=./server-key.pem
--from-file=client.pem=./client.pem
--from-file=client-key.pem=./client-key.pem
6.3 Mount Secrets to Pods
Configure the secrets in the alluxio-cluster.yaml
file under the spec.secrets
section:
For JKS Certificates
spec:
secrets:
coordinator:
alluxio-tls-keystore: /secrets/alluxio-tls-keystore
alluxio-tls-truststore: /secrets/alluxio-tls-truststore
worker:
alluxio-tls-keystore: /secrets/alluxio-tls-keystore
alluxio-tls-truststore: /secrets/alluxio-tls-truststore
For PEM Certificates
spec:
secrets:
coordinator:
my-pem-secret: /alluxio/tls
worker:
my-pem-secret: /alluxio/tls
6.4 Configure Alluxio Properties in Kubernetes
Configure TLS properties in the Alluxio cluster specification. The following examples show configurations for S3 API TLS enablement:
For JKS Certificates
spec:
properties:
# Enable S3 endpoint on workers
alluxio.worker.s3.api.enabled: "true"
# enable s3 TLS
alluxio.worker.s3.tls.enabled: "true"
alluxio.worker.s3.tls.mutual.enabled: "true"
alluxio.worker.s3.https.port: "29996"
alluxio.worker.rest.port: "29998"
alluxio.worker.s3.only.https.access: "false"
alluxio.network.tls.server.protocols: TLSv1.2,TLSv1.3
# Disable the zero-copy
alluxio.worker.s3.local.page.transfer.enabled: "false"
alluxio.worker.network.netty.file.transfer: "MAPPED"
alluxio.network.tls.enabled: "true"
alluxio.network.tls.keystore.path: /secrets/alluxio-tls-keystore/keystore.jks
alluxio.network.tls.keystore.alias: serverkey
alluxio.network.tls.keystore.password: keypass
alluxio.network.tls.keystore.key.password: keypass
alluxio.network.tls.truststore.path: /secrets/alluxio-tls-truststore/truststore.jks
alluxio.network.tls.truststore.password: trustpass
alluxio.network.tls.client.no.endpoint.identification: "true"
For PEM Certificates
spec:
properties:
# Enable S3 endpoint on workers
alluxio.worker.s3.api.enabled: "true"
# enable s3 TLS
alluxio.worker.s3.tls.enabled: "true"
alluxio.worker.s3.tls.mutual.enabled: "true"
alluxio.worker.s3.https.port: "29996"
alluxio.worker.rest.port: "29998"
alluxio.worker.s3.only.https.access: "false"
alluxio.network.tls.server.protocols: TLSv1.2,TLSv1.3
alluxio.network.tls.ca.cert: "/alluxio/tls/ca.pem"
alluxio.network.tls.server.cert: "/alluxio/tls/server.pem"
alluxio.network.tls.server.key: "/alluxio/tls/server-key.pem"
alluxio.network.tls.client.no.endpoint.identification: "true"
# Disable the zero-copy
alluxio.worker.s3.local.page.transfer.enabled: "false"
alluxio.worker.network.netty.file.transfer: "MAPPED"
6.5 Deploy and Restart Services
Apply the configuration and restart services to enable TLS.
Deployment Steps
Apply the configuration:
kubectl apply -f alluxio-cluster.yaml
Restart worker deployment to apply new settings:
kubectl rollout restart deployment alluxio-worker
Verification
After deployment, verify that:
All pods are running successfully
TLS endpoints are accessible
Certificate mounting is working correctly
6.6 Configure TLS for Coordinator-Worker Communication
For enabling TLS between coordinators and workers, add the following to your alluxio-cluster.yaml
:
spec:
properties:
alluxio.network.tls.enabled: "true"
7. Testing and Verification
Comprehensive testing ensures that TLS configuration is working correctly across all components. This section provides testing procedures for different scenarios.
7.1 Testing Overview
Test the following scenarios:
Connectivity: Verify TLS endpoints are accessible
Certificate Validation: Ensure proper certificate chain verification
Security: Confirm non-TLS connections are properly blocked when required
7.2 Verify S3 API TLS Configuration
Test S3 non-TLS (when dual port is enabled):
curl http://<alluxio_worker_svc>:29998/test -v
Test S3 TLS:
curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
https://<alluxio_worker_svc>:29996/test -v
7.3 Verify Gateway TLS Configuration
Test Gateway API endpoints to ensure proper TLS functionality.
Test Gateway non-TLS (when dual port is enabled):
curl -sS 'http://<alluxio_gateway_svc>:80/api/v1/mount' -v
Test Gateway TLS:
curl --http1.1 --cacert ca.pem --cert client.pem --key client-key.pem \
https://<alluxio_gateway_svc>:8443/api/v1/mount' -v
7.4 Verify TLS-Only Configuration
When TLS-only mode is enabled, verify that non-TLS connections are properly rejected:
S3 non-TLS (should fail with connection refused):
curl http://alluxio-worker.default.svc.cluster.local:29998/test -v
Expected output:
* connect to 10.0.5.157 port 29998 failed: Connection refused
* Failed to connect to alluxio-worker.default.svc.cluster.local port 29998: Connection refused
curl: (7) Failed to connect to alluxio-worker.default.svc.cluster.local port 29998: Connection refused
Gateway non-TLS (should fail with connection timeout):
curl -sS 'http://alluxio-gateway.default.svc.cluster.local:80/api/v1/mount' -v
Expected output:
* connect to 172.20.66.21 port 80 failed: Connection timed out
* Failed to connect to alluxio-gateway.default.svc.cluster.local port 80: Connection timed out
curl: (28) Failed to connect to alluxio-gateway.default.svc.cluster.local port 80: Connection timed out
Note: Replace test
with your actual mount point name in the S3 tests.
8. Security Best Practices
Implementing TLS correctly requires attention to certificate management, operational security, and ongoing maintenance. This section outlines industry best practices for TLS deployment.
8.1 Certificate Lifecycle Management
Proper certificate management is crucial for maintaining secure operations.
Certificate Management and Renewal
Regular Certificate Rotation: Implement automated certificate renewal processes
Secure Storage: Store private keys and certificates in secure locations with appropriate file permissions
Backup Strategy: Maintain secure backups of certificates and keys
Key and Certificate Protection
Implement proper security measures for certificate storage and access:
File Permissions: Ensure certificate files are readable only by the Alluxio processes
chmod 600 /alluxio/tls/*.key chmod 644 /alluxio/tls/*.pem
Access Control: Implement proper access controls for certificate directories
Monitoring: Set up monitoring for certificate expiration dates
Multi-Environment Deployment Recommendations
Implement proper isolation and management practices across environments:
Environment Separation: Use different certificates for different environments (dev, staging, production)
Namespace Isolation: In Kubernetes, use different namespaces and secrets for different environments
Configuration Management: Use configuration management tools to maintain consistent TLS settings
9. Appendix
This appendix provides additional resources, command references, and troubleshooting guidance for TLS configuration and management.
9.1 Certificate Format Conversion
Convert between different certificate formats as needed for your environment.
Convert PEM to JKS
# Convert PEM certificate to PKCS12
openssl pkcs12 -export -in server.pem -inkey server-key.pem -out server.p12 -name alluxio
# Convert PKCS12 to JKS
keytool -importkeystore -deststorepass keypass -destkeystore keystore.jks \
-srckeystore server.p12 -srcstoretype PKCS12 -srcstorepass keypass
Convert JKS to PEM
# Export certificate from JKS to PEM
keytool -exportcert -keystore keystore.jks -alias key -rfc -file server.pem
# Export private key (requires conversion through PKCS12)
keytool -importkeystore -srckeystore keystore.jks -destkeystore server.p12 \
-deststoretype PKCS12 -srcalias key -deststorepass keypass -destkeypass keypass
openssl pkcs12 -in server.p12 -nocerts -out server-key.pem -nodes
9.2 Common Commands Reference
Essential commands for certificate management and TLS troubleshooting.
Certificate Information
# View PEM certificate details
openssl x509 -in server.pem -text -noout
# View JKS keystore contents
keytool -list -v -keystore keystore.jks
# Verify certificate chain
openssl verify -CAfile ca.pem server.pem
# Show certificate chain details
openssl crl2pkcs7 -nocrl -certfile ca.pem | openssl pkcs7 -print_certs -noout
openssl crl2pkcs7 -nocrl -certfile server.pem | openssl pkcs7 -print_certs -noout
# Check certificate expiration
openssl x509 -in server.pem -noout -dates
# Validate certificate against CA
openssl x509 -in server.pem -noout -issuer
openssl x509 -in ca.pem -noout -subject
Testing TLS Connectivity
# Test TLS connection
openssl s_client -connect hostname:port -cert client.pem -key client-key.pem -CAfile ca.pem
# Test without client certificate
openssl s_client -connect hostname:port -CAfile ca.pem
# Test with tls protocol (-tls1_2/ -tls1_3)
openssl s_client -connect localhost:29998 -cert client.pem -key client-key.pem -CAfile client-chain.pem -tls1_3
9.3 Troubleshooting Common Issues
Solutions for frequently encountered TLS configuration and connectivity problems.
Certificate Validation Errors
Verify certificate chain and CA certificates
Check certificate expiration dates
Ensure proper file permissions
Connection Failures
Verify port configurations
Check firewall settings
Validate DNS resolution for hostnames
Performance Issues
Monitor the impact of disabling zero-copy transfers
Consider certificate size and algorithms
Evaluate TLS protocol versions
Last updated