Resource Prerequisites and Compatibility
Last updated
Last updated
This guide covers the planning and installation of Alluxio, describing the necessary resource configurations for running Alluxio. It includes resource prerequisites such as CPU, memory, disk, and network requirements. Additionally, it describes compatibility considerations for both the operating system (server, client, and kernel).
Alluxio on Kubernetes (k8s) is the recommended deployment type. The Alluxio coordinator and worker components each require specific CPU, memory, and disk resources. Additionally, the entire Alluxio cluster needs specific network ports to be open for communication. Ensuring the proper configuration of these resources between components is key to achieving efficient operation of Alluxio.
ETCD is also required for service discovery and mounting tables. Please refer to the instructions for to set up ETCD.
In a Kubernetes environment, two sets of configurations are provided for deploying an Alluxio cluster: the minimum configuration and the preferred configuration.
The minimum configuration ensures that the Alluxio cluster can run and support basic IO functional testing.
The preferred configuration utilizes the default settings in the Alluxio operator deployment mode. It optimizes compute, memory, and storage resources for supporting basic Proof of Concept (PoC) scenarios.
These configurations are tailored to accommodate various use cases, offering flexibility and scalability while ensuring stable and efficient operation of the Alluxio cluster within Kubernetes environments.
The table below compares the resources of the Minimum and Preferred configurations for the Alluxio system on K8S deployment.
K8s Node
cpu: 8 CPU cores mem: 32G
cpu: 16 CPU cores mem: 64G
Alluxio coordinator
cpu request: 1 cpu limit: 1 mem request: 1G mem limit: 12G
cpu request: 1 cpu limit: 1 mem request: 8G mem limit: 10G java heap memory: 4G direct memory: 4G
Alluxio worker
cpu request: 1 cpu limit: 10 mem request: 512M mem limit: 20G
cpu request: 1 cpu limit: 10 mem request: 16G mem limit: 20G java heap memory: 8G direct memory: 8G
Page Store
100G
100G, but please adjust based on the number of workers and the actual data usage.
From an architectural perspective, since Alluxio serves as a storage cache, it is recommended to deploy Alluxio components nearby the compute components that run the workload. We recommend deploying Alluxio in the same network. For high performance cases, additional network bandwidth and latency requirements need to be considered.
NOTE: In extreme cases Alluxio may require specialized high performance networking to support high throughput data transfer. Please contact us for detailed information based on your scenarios.
Alluxio on K8s runs on pre-packaged container images and is able to be deployed on any container infrastructure environment including all cloud providers.
Alluxio Cluster
To run Alluxio on Kubernetes, certain prerequisites must be met to ensure compatibility:
Host architecture x86
Kubernetes compatibility with the Host OS or container runtime is beyond the scope of this document. Please refer to and comply with the of your selected Kubernetes distribution or version for detailed information.
A Kubernetes cluster with version at least 1.19, with enabled.