# Cluster Administration This document describes administrative operations on a running Alluxio cluster on Kubernetes, such as upgrading to a new version and adding new workers. ## Dynamically Update Alluxio Configuration in a Running Cluster 1. Get configmap ```console $ kubectl -n alx-ns get configmap NAME DATA AGE alluxio-cluster-alluxio-conf 4 7m48s ... ``` 2. Run `edit configmap` to update Alluxio configuration ```console $ kubectl -n alx-ns edit configmap alluxio-cluster-alluxio-conf ``` There should be 4 files inside: `alluxio-env.sh`, `alluxio-site.properties`, `log4j2.xml`, and `metrics.properties`. Edit as you need, then save the configmap. ```shell configmap/alluxio-cluster-alluxio-conf edited ``` 3. Restart Alluxio components as needed For the cluster `alluxio-cluster` in the `alx-ns` namespace: * coordinator: `kubectl -n alx-ns rollout restart statefulset alluxio-cluster-coordinator` * worker: `kubectl -n alx-ns rollout restart deployment alluxio-cluster-worker` * daemonset fuse (`fuse.type = daemonSet`): `kubectl -n alx-ns rollout restart daemonset alluxio-fuse` * csi fuse (`fuse.type = csi`): the csi fuse pods doesn’t have a `rollout restart` command and must be restarted by either exiting out of the attached user's pod or manually kill it with `kubectl -n alx-ns delete pod alluxio-fuse-xxx`. ## Upgrading to a newer Alluxio version ### Upgrade the Operator 1. Upload the new docker images corresponding to the new Alluxio operator version to your image registry and unpack the helm chart of the operator. Refer to the [installation doc](/ee-ai-en/ai-3.6/start/install/install-alluxio-on-kubernetes.md#preparation) for details. 2. Run the following command to apply the new changes to the cluster. ```shell # uninstall the operator. the operator is independent and the status of the operator won't affect the existing Alluxio cluster $ helm uninstall operator release "operator" uninstalled # check if all the resources are removed. the namespace will be the last resource to remove $ kubectl get ns alluxio-operator Error from server (NotFound): namespaces "alluxio-operator" not found # run the command in the new helm chart directory to upgrade the CRDs first $ kubectl apply -f alluxio-operator/crds 2>/dev/null customresourcedefinition.apiextensions.k8s.io/alluxioclusters.k8s-operator.alluxio.com configured customresourcedefinition.apiextensions.k8s.io/underfilesystems.k8s-operator.alluxio.com configured # use the same operator-config.yaml with only the tag of the image changed to restart the operator $ helm install operator -f operator-config.yaml alluxio-operator NAME: operator LAST DEPLOYED: Thu Jun 27 15:47:44 2024 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ``` ### Upgrade the Alluxio cluster Before the operation you should know: * When the upgrade operation starts, the coordinator, workers, and the DaemonSet FUSE will perform rolling upgrade to use the new image. The existing CSI FUSE pods will not be restarted and upgraded, and only the new pods will use the new image. * While the cluster is being upgraded, the cache hit rate may decrease slightly, but will fully recover once the cluster is fully running again. Following the steps to upgrade the cluster: 1. Upload the new docker images corresponding to the new Alluxio version to your image registry. Refer to the [installation doc](/ee-ai-en/ai-3.6/start/install/install-alluxio-on-kubernetes.md#preparation) for details. 2. Update the `imageTag` fields in `alluxio-cluster.yaml` to reflect the new Alluxio version. In the following example the new `imageTag` will be `AI-3.6-12.0.2`. 3. Run the following command to apply the new changes to the cluster. ```shell # apply the changes to Kubernetes $ kubectl apply -f alluxio-cluster.yaml alluxiocluster.k8s-operator.alluxio.com/alluxio-cluster configured # verify the upgration. you can see the new pods are spawning $ kubectl -n alx-ns get pod NAME READY STATUS RESTARTS AGE alluxio-cluster-coordinator-0 0/1 Init:0/2 0 7s alluxio-cluster-etcd-0 1/1 Running 0 10m alluxio-cluster-etcd-1 1/1 Running 0 10m alluxio-cluster-etcd-2 1/1 Running 0 10m alluxio-cluster-grafana-b89bf9dbb-77pb6 1/1 Running 0 10m alluxio-cluster-prometheus-59b7b8bd64-b95jh 1/1 Running 0 10m alluxio-cluster-worker-58999f8ddd-cd6r2 0/1 Init:0/2 0 7s alluxio-cluster-worker-5d6786f5bf-cxv5j 1/1 Running 0 10m # check the status of the cluster $ kubectl -n alx-ns get alluxiocluster NAME CLUSTERPHASE AGE alluxio-cluster Updating 10m # wait until the cluster is ready again $ kubectl -n alx-ns get alluxiocluster NAME CLUSTERPHASE AGE alluxio-cluster Ready 12m # check the pods of the cluster. you can see the age of the alluxio pods are changed $ kubectl -n alx-ns get pod NAME READY STATUS RESTARTS AGE alluxio-cluster-coordinator-0 1/1 Running 0 93s alluxio-cluster-etcd-0 1/1 Running 0 12m alluxio-cluster-etcd-1 1/1 Running 0 12m alluxio-cluster-etcd-2 1/1 Running 0 12m alluxio-cluster-grafana-b89bf9dbb-77pb6 1/1 Running 0 12m alluxio-cluster-prometheus-59b7b8bd64-b95jh 1/1 Running 0 12m alluxio-cluster-worker-58999f8ddd-cd6r2 1/1 Running 0 93s alluxio-cluster-worker-58999f8ddd-rtftk 1/1 Running 0 33s # double check the version string $ kubectl -n alx-ns exec -it alluxio-cluster-coordinator-0 -- alluxio info version 2>/dev/null AI-3.6-12.0.2 ``` ## Scaling the size of the cluster ### Scale Up the Workers Before the operation you should know: * While the cluster is being upgraded, the cache hit rate may decrease slightly, but will fully recover once the cluster is fully running again. Following the steps to scale up the workers: 1. Change the `alluxio-cluster.yaml` to increase the `count` in the `worker`. In the following example we will scale from 2 workers to 3 workers. 2. Run the following command to apply the new changes to the cluster. ```shell # apply the changes to Kubernetes $ kubectl apply -f alluxio-cluster.yaml alluxiocluster.k8s-operator.alluxio.com/alluxio-cluster configured # verify the cluster is upgrading. you should be able to see the new pods are spawning $ kubectl -n alx-ns get pod NAME READY STATUS RESTARTS AGE alluxio-cluster-coordinator-0 1/1 Running 0 4m51s alluxio-cluster-etcd-0 1/1 Running 0 15m alluxio-cluster-etcd-1 1/1 Running 0 15m alluxio-cluster-etcd-2 1/1 Running 0 15m alluxio-cluster-grafana-b89bf9dbb-77pb6 1/1 Running 0 15m alluxio-cluster-prometheus-59b7b8bd64-b95jh 1/1 Running 0 15m alluxio-cluster-worker-58999f8ddd-cd6r2 1/1 Running 0 4m51s alluxio-cluster-worker-58999f8ddd-rtftk 1/1 Running 0 3m51s alluxio-cluster-worker-58999f8ddd-p6n59 0/1 PodInitializing 0 4s # check if the new instances are ready $ kubectl -n alx-ns get pod NAME READY STATUS RESTARTS AGE alluxio-cluster-coordinator-0 1/1 Running 0 5m21s alluxio-cluster-etcd-0 1/1 Running 0 16m alluxio-cluster-etcd-1 1/1 Running 0 16m alluxio-cluster-etcd-2 1/1 Running 0 16m alluxio-cluster-grafana-b89bf9dbb-77pb6 1/1 Running 0 16m alluxio-cluster-prometheus-59b7b8bd64-b95jh 1/1 Running 0 16m alluxio-cluster-worker-58999f8ddd-cd6r2 1/1 Running 0 5m21s alluxio-cluster-worker-58999f8ddd-rtftk 1/1 Running 0 4m21s alluxio-cluster-worker-58999f8ddd-p6n59 1/1 Running 0 34s ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://documentation.alluxio.io/ee-ai-en/ai-3.6/start/administration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.