Alluxio supports high availability by leveraging multiple availability zones. An Alluxio client can fall back to workers in other AZs when all the workers from its local AZ fails to serve the requests.
The multi-AZ support, along with multiple replication and Under File System fallback, provides strong service availability and I/O resiliency. You can read more about the features in Alluxio.
Enabling Multi-Availability Zone support
A typical deployment when multiple AZs are available, is to deploy one Alluxio cluster per AZ. To differentiate the clusters in each AZ, each Alluxio cluster must be assigned a unique cluster name.
Create a JSON configuration file to specify all the Alluxio clusters in different AZs with their respective etcd clusters. The following is an example with 3 clusters:
Save the configuration in Alluxio's configuration directory as multi-az-clusters.json and link to it in the Alluxio configuration file alluxio-site.properties:
If Alluxio is already running, restart all processes to apply the new configuration. After the restart, check with the following command to see if multi-AZ support is working as expected:
$ bin/alluxio info nodes
When configured properly, this command will report the information of the Alluxio worker nodes from all the clusters in different AZs:
Deploying Multiple Alluxio Clusters Using the Operator
The Operator supports deploying multiple Alluxio clusters through the use of a clusterGroup resource.
This custom resource enables users to define and manage the configuration of multiple Alluxio clusters consistently across a Kubernetes environment.
By leveraging the clusterGroup, users can efficiently orchestrate and maintain multiple Alluxio clusters with a shared configuration.
Note: Clusters created through a clusterGroup share the same configuration, including properties, resource allocation and scale.
To differentiate clusters, the nodeSelector field is used—a Kubernetes mechanism that constrains Pods to be scheduled
on specific nodes. Each Alluxio cluster should be assigned to distinct AZ using nodeSelector
to ensure proper separation and deployment.
Preparation Steps
This example will deploy three Alluxio clusters across two Kubernetes namespaces:
Cluster 1: alluxio-a in namespace alx-ns
Cluster 2: alluxio-b in namespace alx-ns
Cluster 3: alluxio-c in namespace alx-ns-2
Ensure the necessary namespaces (alx-ns and alx-ns-2) are created prior to deploying the clusters:
A ConfigMap containing the cluster names and ETCD info of each cluster needs to be created to configure the clusters. Depending on the configuration of ETCD, the ETCD info of each cluster could differ. When deploying multiple Alluxio clusters using the clusterGroup resource, there are three supported deployment modes, categorized by the ETCD setup:
Independent ETCD for Each Cluster
Each Alluxio cluster uses its own dedicated ETCD instance.
Shared ETCD Across Clusters
Multiple clusters share a single ETCD instance while maintaining isolated namespaces.
External ETCD Integration
Clusters connect to an externally managed ETCD service.
These modes provide flexibility to accommodate different operational requirements and infrastructure architectures. Users can select the appropriate mode.
Independent ETCD Mode
In the Independent ETCD deployment mode, each Alluxio cluster operates with its own dedicated ETCD instance.
In this mode, the configuration file multi-az-clusters.json must be customized to specify individual ETCD endpoints for each cluster, as shown below:
In this configuration, each Alluxio cluster will independently start and manage its own ETCD instance within its respective namespace. The clusterGroup itself does NOT deploy an ETCD cluster.
You can verify this by listing the pods in the cluster group namespace:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
alluxio-cg-dashboard-dfd4dcfb5-fvj8j 1/1 Running 0 3h27m
alluxio-cg-gateway-59df98fb66-kkz6l 1/1 Running 0 3h27m
Each Alluxio cluster will run its own ETCD service in its corresponding namespace. For example, in namespace alx-ns, both alluxio-a and alluxio-b clusters deploy separate ETCD pods:
In the shared ETCD deployment mode, all Alluxio clusters share a single ETCD cluster for coordination.
Start by creating a JSON configuration file named multi-az-clusters.json, which specifies the participating Alluxio clusters and their shared ETCD endpoints:
In External ETCD mode, all Alluxio clusters are connected to a shared, externally managed ETCD cluster.
If you already have an external ETCD cluster deployed, you can configure Alluxio clusters to connect to it:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
external-etcd-0 1/1 Running 0 6m10s
external-etcd-1 1/1 Running 0 6m10s
external-etcd-2 1/1 Running 0 6m10s
Create a configuration file multi-az-clusters.json to define the shared external ETCD endpoint across all clusters:
Enabling optimized I/O for multi-AZ replicated files
For files that are not replicated in multiple AZs, the client will not fall back to other AZs when the local workers fail to serve the request, and will fall back to the UFS directly.
Note that if this feature is enabled, a client prefers a fully cached replica over a partially cached one, even if it is from a remote worker. The client will choose a preferred data source according to the following order:
A local worker that has fully cached the file.
A remote worker that has fully cached the file.
When no workers, local or remote, have fully cached the file, a local worker.
When no local candidate workers are available, a remote worker.
When no candidate workers in any AZ are available, the UFS.
Enabling passive cache
To enable optimized I/O and passive cache, add the following configuration to alluxio-site.properties:
# Note that multi-replica optimized IO must be enabled for passive cache to take effect
alluxio.user.replica.prefer.cached.replicas=true
alluxio.user.file.passive.cache.enabled=true
The Alluxio Operator provides a standardized method for deploying and managing Alluxio clusters within a Kubernetes environment. For comprehensive instructions on installing the Alluxio Operator,
please refer to the .
Alluxio supports . In the case of files that are replicated in multiple AZs, in addition to faster access, the replications can also provide high availability when some clusters fail due to an AZ outage.
When an Alluxio client reads a multi-AZ replicated file, it will first check its local cluster to see if the file is cached by any of the local workers. If so, the client will try to use a local worker to read the file. If none of the local workers have cached the file, or the client encounters an error reading from all the local workers in the case of an outage in the local AZ, it will fall back to workers in other clusters in other AZs. If all the candidate workers fail to serve the read request, the client will eventually fall back to the UFS as a last resort.
It is recommended to enable
in a multi-AZ replicated deployment. This ensures under-replicated files will automatically get more replicas and
the performance will not be impacted because of cache misses on the preferred workers.
