Kubernetes
Prerequisites
A Kubernetes cluster (version 1.11+) with Beta feature gate APIs enabled
The Alluxio Helm chart which the Kubernetes resource specifications are built from supports Kubernetes version 1.11+.
Beta feature gates are enabled by default for Kubernetes cluster installations
Cluster access to an Alluxio Docker image alluxio/alluxio-enterprise. If using a private Docker registry, refer to the Kubernetes private image registry documentation.
Ensure the cluster's Kubernetes Network Policy allows for connectivity between applications (Alluxio clients) and the Alluxio Pods on the defined ports.
Basic Setup
This tutorial walks through a basic Alluxio setup on Kubernetes. Alluxio supports two methods of installation on Kubernetes: either using helm charts or using kubectl. When available, helm is the preferred way to install Alluxio. If helm is not available or if additional deployment customization is desired, kubectl can be used directly using native Kubernetes resource specifications.
(Optional) This step is not required if using the trial edition docker image available on dockerhub. If not using the trial edition, load the Alluxio Docker image from the tar and host in a private docker registry. Refer to the Kubernetes documentation.
docker load --input alluxio-enterprise-alluxio-2.7.0-docker.tarNote: From Alluxio 2.3 on, Alluxio only supports helm 3. See how to migrate from helm 2 to 3 here.
Deploy
Verify
If using persistent volumes, the status of the volume(s) should change to CLAIMED and the status of the volume claims should be BOUNDED. You can validate the status of your PersistentVolume and PersistentVolumeClaims using the follow kubectl commands:
$ kubectl get pv
$ kubectl get pvcIf you have unbound PersistentVolumeClaims, please ensure you have provisioned matching PersistentVolumes. See "(Optional) Provision a Persistent Volume" in Basic Setup.
Once ready, access the Alluxio CLI from the master Pod and run basic I/O tests.
$ kubectl exec -ti alluxio-master-0 /bin/bashFrom the master Pod, execute the following:
$ alluxio runTestsAccess the Web UI
The Alluxio UI can be accessed from outside the kubernetes cluster using port forwarding.
$ kubectl port-forward alluxio-master-$i <local-port>:19999The command above allocates a port on the local node <local-port> and forwards traffic on <local-port> to port 19999 of pod alluxio-master-$i. The pod alluxio-master-$i does NOT have to be on the node you are running this command.
Note:
i=0for the the first master Pod. When running multiple masters, forward port for each master. Only the primary master serves the Web UI.
For example, you are on a node with hostname master-node-1 and you would like to serve the Alluxio master web UI for alluxio-master-0 on master-node-1:8080. Here's the command you can run:
[alice@master-node-1 ~]$ kubectl port-forward --address 0.0.0.0 pods/alluxio-master-0 8080:19999This forwards the local port master-node-1:8080 to the port on the Pod alluxio-master-0:19999. The Pod alluxio-master-0 does NOT need to be running on master-node-1.
You will see messages like below when there are incoming connections.
[alice@master-node-1 ~]$ kubectl port-forward --address 0.0.0.0 alluxio-master-0 8080:19999
Forwarding from 0.0.0.0:8080 -> 19999
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080You can terminate the process to stop the port forwarding, with either Ctrl + C or kill.
For more information about K8s port-forward see the K8s doc.
Advanced Setup
Enable Short-circuit Access
Short-circuit access enables clients to perform read and write operations directly against the worker bypassing the networking interface. For performance-critical applications it is recommended to enable short-circuit operations against Alluxio because it can increase a client's read and write throughput when co-located with an Alluxio worker.
This feature is enabled by default (see next section to disable this feature), however requires extra configuration to work properly in Kubernetes environments.
There are two modes for using short-circuit.
Option1: Use local mode
In this mode, the Alluxio client and local Alluxio worker recognize each other if the client hostname matches the worker hostname. This is called Hostname Introspection. In this mode, the Alluxio client and local Alluxio worker share the tiered storage of Alluxio worker.
Option2: Use uuid (default)
This is the default policy used for short-circuit in Kubernetes.
If the client or worker container is using virtual networking, their hostnames may not match. In such a scenario, set the following property to use filesystem inspection to enable short-circuit operations and make sure the client container mounts the directory specified as the domain socket path. Short-circuit writes are then enabled if the worker UUID is located on the client filesystem.
Domain Socket Path. The domain socket is a volume which should be mounted on:
All Alluxio workers
All application containers which intend to read/write through Alluxio
This domain socket volume can be either a PersistentVolumeClaim or a hostPath Volume.
Use PersistentVolumeClaim. By default, this domain socket volume is a PersistentVolumeClaim. You need to provision a PersistentVolume to this PersistentVolumeClaim. And this PersistentVolume should be either local or hostPath.
Use hostPath Volume. You can also directly define the workers to use a hostPath Volume for domain socket.
Verify Short-circuit Operations
To verify short-circuit reads and writes monitor the metrics displayed under:
the metrics tab of the web UI as
Domain Socket Alluxio ReadandDomain Socket Alluxio Writeor, the metrics json as
cluster.BytesReadDomainandcluster.BytesWrittenDomainor, the fsadmin metrics CLI as
Short-circuit Read (Domain Socket)andAlluxio Write (Domain Socket)
Disable Short-Circuit Operations
To disable short-circuit operations, the operation depends on how you deploy Alluxio.
Note: As mentioned, disabling short-circuit access for Alluxio workers will result in worse I/O throughput
Enable remote logging
Alluxio supports a centralized log server that collects logs for all Alluxio processes. You can find the specific section at Remote logging. This can be enabled on K8s too, so that all Alluxio pods will send logs to this log server.
Verify log server
You can go into the log server pod and verify the logs exist.
$ kubectl exec -it <logserver-pod-name> bash
# In the logserver pod
bash-4.4$ pwd
/opt/alluxio
# You should see logs collected from other Alluxio pods
bash-4.4$ ls -al logs
total 16
drwxrwsr-x 4 1001 bin 4096 Jan 12 03:14 .
drwxr-xr-x 1 alluxio alluxio 18 Jan 12 02:38 ..
drwxr-sr-x 2 alluxio bin 4096 Jan 12 03:14 job_master
-rw-r--r-- 1 alluxio bin 600 Jan 12 03:14 logserver.log
drwxr-sr-x 2 alluxio bin 4096 Jan 12 03:14 master
drwxr-sr-x 2 alluxio bin 4096 Jan 12 03:14 worker
drwxr-sr-x 2 alluxio bin 4096 Jan 12 03:14 job_workerPOSIX API
Once Alluxio is deployed on Kubernetes, there are multiple ways in which a client application can connect to it. For applications using the POSIX API, application containers can simply mount the Alluxio FileSystem.
In order to use the POSIX API, first deploy the Alluxio FUSE daemon.
Toggle Master or Worker in Helm chart
In use cases where you wish to install Alluxio masters and workers separately with the Helm chart, use the following respective toggles:
master:
enabled: false
worker:
enabled: falseKubernetes Configuration Options
The following options are provided in our Helm chart as additional parameters for experienced Kubernetes users.
ServiceAccounts
By default Kubernetes will assign the namespace's default ServiceAccount to new pods in a namespace. You may specify for Alluxio pods to use any existing ServiceAccounts you may have in your cluster through the following:
Node Selectors & Tolerations
Kubernetes provides many options to control the scheduling of pods onto nodes in the cluster. The most direct of which is a node selector.
However, Kubernetes will avoid scheduling pods on any tainted nodes. To allow certain pods to schedule on such nodes, Kubernetes allows you to specify tolerations for those taints. See the Kubernetes documentation on taints and tolerations for more details.
Host Aliases
If you wish to add or override hostname resolution in the pods, Kubernetes exposes the containers' /etc/hosts file via host aliases. This can be particularly useful for providing hostname addresses for services not managed by Kubernetes, like HDFS.
Deployment Strategy
By default Kubernetes will use the 'RollingUpdate' deployment strategy to progressively upgrade Pods when changes are detected.
ImagePullSecrets
Kubernetes supports accessing images from a Private Registry. After creating the registry credentials Secret in Kubernetes, you pass the secret to your Pods via imagePullSecrets.
Troubleshooting
Last updated