Install on Kubernetes
This documentation shows how to install Alluxio on Kubernetes via Operator, a Kubernetes extension for managing applications.
Prerequisites
Kubernetes
A Kubernetes cluster with version at least 1.19, with feature gates enabled
Ensure the cluster’s Kubernetes Network Policy allows for connectivity between applications (Alluxio clients) and the Alluxio Pods on the defined ports
The Kubernetes cluster must have Helm 3 installed, with a version of 3.6.0 or higher
Image registry for storing and managing container images
Hardware
Each node should have at least 8 CPUs and 32GB of memory
Each node should have at least 100GB of storage space
Permissions. Reference: Using RBAC Authorization
Permission to create CRD (Custom Resource Definition)
Permission to create ServiceAccount, ClusterRole, and ClusterRoleBinding for the operator pod
Permission to create namespace that the operator will be in
Preparation
Download the files for Alluxio operator and Alluxio cluster
Request a trial version of Alluxio enterprise DA by contacting your Alluxio account representative at [email protected]
.
Follow their instructions to download the files into the directory you prepared.
Obtain the following files:
alluxio-operator-3.0.2-helmchart.tgz
is the helm chart for deploying Alluxio operatoralluxio-operator-3.0.2-docker.tar
is the docker image for all Alluxio operator componentsalluxio-enterprise-DA-3.5-10.2.0-docker.tar
is the docker image for Alluxioalluxio-enterprise-DA-3.5-10.2.0-release.tar.gz
is a tarball containing jars to incorporate with specific compute integrations to enable them to connect to the Alluxio cluster This file will not be used within this document's instructions, but will be expected to be available when preparing the compute cluster.
Also obtain the license string referred to as <ALLUXIO_LICENSE_STRING>
in later sections.
Upload the images to an image registry
An image registry is a centralized location for storing and sharing your container images. It can be either public or private. Cloud providers generally provide container registries as a service, such as Amazon Elastic Container Registry(ECR),Azure Container Registry (ACR), and Google Container Registry (GCR). Private registries may also be provided from your local system or within private networks of your organization.
This example shows how to upload the Alluxio operator images.
# load the image to local
$ docker load -i alluxio-operator-3.0.2-docker.tar
$ docker load -i alluxio-enterprise-DA-3.5-10.2.0-docker.tar
# retag the image with your private registry
$ docker tag alluxio/operator:3.0.2 <PRIVATE_REGISTRY>/alluxio-operator:3.0.2
$ docker tag alluxio/alluxio-enterprise:DA-3.5-10.2.0 <PRIVATE_REGISTRY>/alluxio-enterprise:DA-3.5-10.2.0
# push to the remote registry
$ docker push <PRIVATE_REGISTRY>/alluxio-operator:3.0.2
$ docker push <PRIVATE_REGISTRY>/alluxio-enterprise:DA-3.5-10.2.0
Extract the helm chart for operator
# the command will extract the files to the directory alluxio-operator/
$ tar zxf alluxio-operator-3.0.2-helmchart.tgz
The extracted alluxio-operator
directory contains the Helm chart files responsible for deploying the operator.
Deployment
Deploy Alluxio operator
Create the alluxio-operator/alluxio-operator.yaml
file to specify the image and version used for deploying the operator.
The following example shows how to specify the operator
image and version:
global:
image: <PRIVATE_REGISTRY>/alluxio-operator
imageTag: 3.0.2
alluxio-csi:
enabled: false
Move to the alluxio-operator
directory and execute the following command to deploy the operator:
$ cd alluxio-operator
# the last parameter is the directory to the helm chart, "." means the current directory
$ helm install operator -f alluxio-operator.yaml .
NAME: operator
LAST DEPLOYED: Wed May 15 17:32:34 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
# verify if the operator is running as expected
$ kubectl get pod -n alluxio-operator
NAME READY STATUS RESTARTS AGE
alluxio-cluster-controller-5647cc664d-lrx84 1/1 Running 0 14s
alluxio-collectinfo-controller-667b746fd6-hfzqk 1/1 Running 0 14s
alluxio-ufs-controller-5f6d7c4d66-drjgm 1/1 Running 0 14s
Deploy Alluxio
Create the alluxio-operator/alluxio-cluster.yaml
file to deploy the Alluxio cluster. The file below shows the minimal configuration.
Please see Resource Prerequisites and Compatibility for resource planning recommendations.
The operator has set the recommended configuration for production deployment by default.
If you need to modify the configuration, you can edit the .spec.properties
field in the alluxio-cluster.yaml
file.
The section for common use cases describes some general scenarios to modify these properties.
The properties specified under the
.spec.properties
field will be appended to thealluxio-site.properties
configuration file and the Alluxio processes will read that file. You can find your configurations in the Alluxio coordinator or worker pods by looking at/opt/alluxio/conf/alluxio-site.properties
.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
metadata:
name: alluxio
spec:
image: <PRIVATE_REGISTRY>/alluxio-enterprise
imageTag: DA-3.5-10.2.0
properties:
alluxio.license: <ALLUXIO_LICENSE_STRING>
worker:
count: 2
pagestore:
size: 100Gi
reservedSize: 10Gi
fuse:
type: none
Note that by default the page store is located at the host path /mnt/alluxio/pagestore
. The size of the cache can be
set by the .spec.pagestore.size
property. The size of the reserved space can be set by the .spec.pagestore.reservedSize
property, and is recommended to be 5% - 10% of the cache size. Please adjust the size of the cache and the reserved space
according to the actual capacity of the storage device of the host.
If your data is stored in a storage such as S3 and the compute task can access the data via
s3://
, you can accelerate the workload by mounting the under file system into Alluxio through creating UFS resources after starting the Alluxio cluster.
Move to the alluxio-operator
directory and execute the following command to deploy the Alluxio cluster:
$ cd alluxio-operator
$ kubectl create -f alluxio-cluster.yaml
alluxiocluster.k8s-operator.alluxio.com/alluxio created
# the cluster will be starting
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
alluxio-coordinator-0 0/1 Init:0/1 0 7s
alluxio-etcd-0 0/1 ContainerCreating 0 7s
alluxio-etcd-1 0/1 ContainerCreating 0 7s
alluxio-etcd-2 0/1 ContainerCreating 0 7s
alluxio-grafana-847fd46f4b-84wgg 0/1 Running 0 7s
alluxio-prometheus-778547fd75-rh6r6 1/1 Running 0 7s
alluxio-worker-76c846bfb6-2jkmr 0/1 Init:0/2 0 7s
alluxio-worker-76c846bfb6-nqldm 0/1 Init:0/2 0 7s
# check the status of the cluster
$ kubectl get alluxiocluster
NAME CLUSTERPHASE AGE
alluxio Ready 2m18s
# and check the running pods after the cluster is ready
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
alluxio-coordinator-0 1/1 Running 0 2m3s
alluxio-etcd-0 1/1 Running 0 2m3s
alluxio-etcd-1 1/1 Running 0 2m3s
alluxio-etcd-2 1/1 Running 0 2m3s
alluxio-grafana-7b9477d66-mmcc5 1/1 Running 0 2m3s
alluxio-prometheus-78dbb89994-xxr4c 1/1 Running 0 2m3s
alluxio-worker-85fd45db46-c7n9p 1/1 Running 0 2m3s
alluxio-worker-85fd45db46-sqv2c 1/1 Running 0 2m3s
In Alluxio 3.x, the coordinator is a stateless control component that serves jobs like distributed load and acts as API gateway for the operator.
If some components in the cluster do not reach the
Running
state, you can usekubectl describe pod
to view detailed information and identify the issue. For specific issues encountered during deployment, refer to the FAQ section.
Alluxio cluster also includes etcd and monitoring components. If the image cannot be pulled from the public image registry, causing etcd and monitoring to fail to start, please refer to the section for unable to access public image registry.
Mount storage to Alluxio
Alluxio supports integration with various underlying storage systems, including S3 and HDFS.
With the operator, you can mount underlying storage by creating UnderFileSystem
resources.
An UnderFileSystem
corresponds to a mount point for Alluxio.
Regarding the Alluxio and the underlying storage namespace,
please refer to Alluxio Namespace and Under File System Namespaces.
Below, we provide an examples of using S3 as the underlying storage mount.
Create the alluxio-operator/ufs.yaml
file to specify the UFS configuration. The following example shows how to mount an S3 bucket to Alluxio:
apiVersion: k8s-operator.alluxio.com/v1
kind: UnderFileSystem
metadata:
name: alluxio-s3
spec:
alluxioCluster: alluxio
path: s3://my-bucket/path/to/mount
mountPath: /s3
mountOptions:
s3a.accessKeyId: <YOUR_S3_ACCESS_KEY>
s3a.secretKey: <YOUR_S3_SECRET_KEY>
alluxio.underfs.s3.region: us-east-1
Find more details about mounting S3 to Alluxio in Amazon AWS S3.
Executing the mount
First, ensure that the Alluxio cluster is up and running with a Ready
status.
(Or if the status is WaitingForReady
, can also mount UFS)
# check the status of the cluster
$ kubectl get alluxiocluster
NAME CLUSTERPHASE AGE
alluxio Ready 2m18s
Execute the following command to create the UnderFileSystem
resource and mount that to Alluxio namespace:
$ cd alluxio-operator
$ kubectl create -f ufs.yaml
underfilesystem.k8s-operator.alluxio.com/alluxio-s3 created
# verify the status of the storage
$ kubectl get ufs
NAME PHASE AGE
alluxio-s3 Ready 46s
# also check the mount table via Alluxio command line
$ kubectl exec -it alluxio-coordinator-0 -- alluxio mount list 2>/dev/null
Listing all mount points
s3://my-bucket/path/to/mount on /s3/ properties={s3a.secretKey=xxx, alluxio.underfs.s3.region=us-east-1, s3a.accessKeyId=xxx}
Monitoring
The Alluxio cluster enables monitoring by default. You can view various Alluxio metrics visually through Grafana. Please refer to the Monitoring and Metrics section on Kubernetes Operator.
Common use cases
Change the resource limitations
For every component, like worker and coordinator, we can change the resource by the following configuration:
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
worker:
count: 2
resources:
limits:
cpu: "12"
memory: "36Gi"
requests:
cpu: "1"
memory: "32Gi"
jvmOptions:
- "-Xmx22g"
- "-Xms22g"
- "-XX:MaxDirectMemorySize=10g"
coordinator:
resources:
limits:
cpu: "12"
memory: "36Gi"
requests:
cpu: "1"
memory: "32Gi"
jvmOptions:
- "-Xmx4g"
- "-Xms1g"
The container will never be able to access the resource over the limits, and the requests are used during scheduling. For more information, please refer to Resource Management for Pods and Containers
The limit of the memory should be a little over the sum of the heap size(
-Xmx
) and the direct memory size(-XX:MaxDirectMemorySize=10g
) to avoid out-of-memory problems.For demo purpose, recommend to set smaller resource limits, for example
worker:
resources:
limits:
cpu: "8"
memory: "4Gi"
requests:
cpu: "1"
memory: "1Gi"
Use PVC for page store
Page store here refer to the cache Alluxio uses.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
worker:
pagestore:
type: persistentVolumeClaim
storageClass: ""
size: 100Gi
reservedSize: 10Gi
The PVC will be created by the operator
The
storageClass
defaults tostandard
, but can be specified to empty string for static bindingThe
size
property specifies the size of the cache space. ThereservedSize
property specifies the amount of additional space used as an internal buffer for temporary data. The total size of the underlying storage will be the sum of the size of the cache and the reserved size. We recommend allocating a reserved size that's 5% - 10% of the size of the cache.
Mount customized config maps
A custom config map can be used to provide configuration files on pods. Although it can be used for other purposes such as environment variables, the following example will focus on files.
Create a new config map from a local file:
kubectl create configmap my-configmap --from-file=/path/to/my-configmap
Declare the config map with its mount point.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
configMaps:
worker:
my-configmap: /path/to/mount
coordinator:
my-configmap: /path/to/mount
The key is the name of the
ConfigMap
, and the value if the mounted path in the containerThe
/opt/alluxio/conf
is already mounted by default. This means other files cannot be mounted directly within theconf/
directory. The custom config maps need to mount to other paths.Using the cache filter json file as an example, mount it to
/opt/alluxio/conf/cachefilter/cache_filter.json
and set this path as the value ofalluxio.user.client.cache.filter.config.file
for Alluxio to read it.
Add a file onto pods as a secret
This mechanism can be used to provide credentials files on pods.
Create a new secret from a local file:
kubectl create secret my-file --from-file=/path/to/my-file
Specify which secrets to load and the file path on the pods.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
secrets:
worker:
my-file: /home/alluxio/my-file
coordinator:
my-file: /home/alluxio/my-file
Use the root user
The processes use the user with uid 1000 by default. In the container, the user is named alluxio
.
To change it to the root user, use this configuration:
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
user: 0
group: 0
fsGroup: 0
Sometimes it’s enough to specify the
.spec.fsGroup = 0
only when the files can be accessed by the root groupThe ownership of the mounted host path, such as the page store path and log path, will be transferred to root if changing to the root user.
Use external ETCD
If you have an external ETCD cluster, you can specify the endpoints for Alluxio to use.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
etcd:
enabled: false
properties:
alluxio.etcd.endpoints: http://external-etcd:2379
When Client-to-server transport security with HTTPS, certificates are used for SSL/TLS connections to ETCD.
For this, have a signed key pair (client.crt
, pkcs8_key_encrypted.pem
) and CA file (ca.crt
) ready.
Here it needs a PKCS8 key, you can use the following command to convert the key:
$ openssl pkcs8 -topk8 -v2 aes256 -in server.key -out pkcs8_key_encrypted.pem
Note: If you use openssl pkcs8 -topk8 -nocrypt -in server.key -out pkcs8_key.pem
generate unencrypted key file. You don’t need to set alluxio.etcd.tls.client.key.password
in alluxio-site.properties.
Create secrets in Kubernetes with the created ca.crt
, client.crt
and pkcs8_key_encrypted.pem
. For example,
$ kubectl create secret generic etcd-certs --from-file=/path/to/ca.crt --from-file=/path/to/client.crt --from-file=/path/to/pkcs8_key_encrypted.pem
Configure the etcd properties in the alluxio-cluster.yaml
file and specify the secrets for the coordinator and worker:
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
etcd:
enabled: false
properties:
alluxio.etcd.endpoints: https://external-etcd:2379
alluxio.etcd.tls.enabled: "true"
alluxio.etcd.tls.ca.cert: /secrets/etcd-certs/ca.crt
alluxio.etcd.tls.client.cert: /secrets/etcd-certs/client.crt
alluxio.etcd.tls.client.key: /secrets/etcd-certs/pkcs8_key_encrypted.pem
alluxio.etcd.tls.client.key.password: <your key password>
secrets:
coordinator:
etcd-certs: /secrets/etcd-certs
worker:
etcd-certs: /secrets/etcd-certs
Deploy workers on nodes with different disk specs
The operator supports heterogeneous configurations for workers, specifically to configure different disk specs. Generally, inconsistencies within the worker configurations may lead to serious unexpected errors and therefore we do not support other scenarios other than the following use case.
Classify the nodes with the disk specs. e.g.: We have 10 nodes with one 1TB disk and 12 nodes with two 800GB disks.
Label the nodes to uniquely identify different groups of workers, where each group shares the same configuration.
# label nodes with one disk
kubectl label nodes <node name> apps.alluxio.com/disks=1
# label nodes with two disks
kubectl label nodes <node name> apps.alluxio.com/disks=2
Use
.workerGroups
to list the worker configurations, defining the labels to filter on withnodeSelector
and its corresponding configuration.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
# you can still specify common configurations with .worker
worker:
# the resources and the jvmOptions will affect all worker groups
resources:
limits:
memory: 40Gi
requests:
memory: 36Gi
jvmOptions: ["-Xmx20g", "-Xms20g", "-XX:MaxDirectMemorySize=16g"]
# configuration here will override the one in worker
workerGroups:
- worker:
count: 10
nodeSelector:
apps.alluxio.com/disks: 1
pagestore:
hostPath: /mnt/disk1/alluxio/pagestore
size: 1Ti
- worker:
count: 12
nodeSelector:
apps.alluxio.com/disks: 2
pagestore:
hostPath: /mnt/disk1/alluxio/pagestore,/mnt/disk2/alluxio/pagestore
size: 800Gi,800Gi
Dynamically Update Alluxio Configuration in a Running Cluster
Get configmap
$ kubectl get configmap
NAME DATA AGE
alluxio-alluxio-conf 4 7m48s
Edit configmap to update Alluxio configuration
$ kubectl edit configmap alluxio-alluxio-conf
There should be 4 files inside: alluxio-env.sh
, alluxio-site.properties
, log4j2.xml
, and metrics.properties
.
Edit as you need, then save the configmap.
configmap/alluxio-alluxio-conf edited
Restart Alluxio components as needed Assuming the cluster name is
alluxio
which is specified inalluxio-cluster.yaml
coordinator:
kubectl rollout restart statefulset alluxio-coordinator
worker:
kubectl rollout restart deployment alluxio-worker
FAQ
etcd pod stuck in pending status
For example, if three etcd pods remain in the Pending state, you can use kubectl describe pod
to view detailed information:
# Check the status of the pods
kubectl get pod
NAME READY STATUS RESTARTS AGE
alluxio-coordinator-0 0/1 Init:1/2 0 73s
alluxio-etcd-0 0/1 Pending 0 73s
alluxio-etcd-1 0/1 Pending 0 73s
alluxio-etcd-2 0/1 Pending 0 73s
alluxio-grafana-79db8c7dd9-lsq2l 1/1 Running 0 73s
alluxio-prometheus-7c6cbc4b4c-9nk25 1/1 Running 0 73s
alluxio-worker-8c79d5fd4-2c994 0/1 Init:1/2 0 73s
alluxio-worker-8c79d5fd4-jrchj 0/1 Init:1/2 0 73s
# Check detailed information about the etcd pod
kubectl describe pod alluxio-etcd-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m57s default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling., .
# Check the PVC Status in the Cluster
# If you find that the etcd PVCs are stuck in the Pending state, you can investigate further.
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-alluxio-etcd-0 Pending 3h41m
data-alluxio-etcd-1 Pending 3h41m
data-alluxio-etcd-2 Pending 3h41m
# Check the PVC description
kubectl describe pvc data-alluxio-etcd-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4m16s (x889 over 3h44m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Based on the error message, the etcd pods are stuck in the Pending state because no storage class is set.
You can resolve this issue by specifying the storage class for etcd in the alluxio-operator/alluxio-cluster.yaml
file:
etcd:
persistence:
storageClass: <STORAGE_CLASS>
size:
First, delete the Alluxio cluster and the etcd PVC, then recreate the Alluxio cluster:
# Delete the Alluxio cluster
$ kubectl delete -f alluxio-operator/alluxio-cluster.yaml
# Delete the etcd PVC
$ kubectl delete pvc data-alluxio-etcd-0
$ kubectl delete pvc data-alluxio-etcd-1
$ kubectl delete pvc data-alluxio-etcd-2
# Recreate the Alluxio cluster
$ kubectl create -f alluxio-operator/alluxio-cluster.yaml
Another issue is the etcd PVC specifies a storage class, but both the etcd pod and PVC remain in a pending state. For example, as shown in the detailed information of the PVC below, the storage class specified for the etcd PVC does not support dynamic provisioning, and the storage volume needs to be manually created by the cluster administrator.
# Check the PVC description
kubectl describe pvc data-alluxio-etcd-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 25s persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 8s (x3 over 25s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'none' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
For similar issues where etcd pods remain in the Pending state, you can use the above method for troubleshooting.
Unable to access public image registry
Starting the Alluxio cluster also involves etcd and monitoring components. If you cannot access the public image registry, you need to provide the images for these components from your private image registry.
You can download the dependent images locally, upload them to your private image registry,
and then modify the image addresses in the alluxio-operator.yaml
file before redeploying the operator.
cluster ETCD
docker.io/bitnami/etcd
3.5.9-debian-11-r24
etcd dependency
cluster ETCD
docker.io/bitnami/os-shell
11-debian-11-r2
os-shell dependency
cluster monitor
grafana/grafana
10.4.5
Monitoring dashboard
cluster monitor
prom/prometheus
v2.52.0
Metrics collection
The commands to pull the Docker images and upload them to your private image registry are as follows:
# Pull the Docker images
$ docker pull docker.io/bitnami/etcd:3.5.9-debian-11-r24
$ docker pull docker.io/bitnami/os-shell:11-debian-11-r2
$ docker pull grafana/grafana:10.4.5
$ docker pull prom/prometheus:v2.52.0
# Tag the images with your private registry
$ docker tag docker.io/bitnami/etcd:3.5.9-debian-11-r24 <PRIVATE_REGISTRY>/etcd:3.5.9-debian-11-r24
$ docker tag docker.io/bitnami/os-shell:11-debian-11-r2 <PRIVATE_REGISTRY>/os-shell:11-debian-11-r2
$ docker tag grafana/grafana:10.4.5 <PRIVATE_REGISTRY>/grafana:10.4.5
$ docker tag prom/prometheus:v2.52.0 <PRIVATE_REGISTRY>/prometheus:v2.52.0
# Push the images to your private registry
$ docker push <PRIVATE_REGISTRY>/etcd:3.5.9-debian-11-r24
$ docker push <PRIVATE_REGISTRY>/os-shell:11-debian-11-r2
$ docker push <PRIVATE_REGISTRY>/grafana:10.4.5
$ docker push <PRIVATE_REGISTRY>/prometheus:v2.52.0
Modify the image addresses in the alluxio-operator/alluxio-cluster.yaml
file accordingly.
apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
metadata:
name: alluxio
spec:
image: <PRIVATE_REGISTRY>/alluxio-enterprise
imageTag: DA-3.5-10.2.0
properties:
worker:
count: 2
pagestore:
size: 100Gi
reservedSize: 10Gi
etcd:
image:
registry: <PRIVATE_REGISTRY>
repository: <PRIVATE_REPOSITORY>/etcd
tag: 3.5.9-debian-11-r24
volumePermissions:
image:
registry: <PRIVATE_REGISTRY>
repository: <PRIVATE_REPOSITORY>/os-shell
tag: 11-debian-11-r2
prometheus:
image: <PRIVATE_REGISTRY>/prometheus
imageTag: v2.52.0
grafana:
image: <PRIVATE_REGISTRY>/grafana
imageTag: 10.4.5
Last updated