Advanced Configuration

The operator has already set the recommended configuration by default, which can start an Alluxio cluster. The following sections describe common examples and use cases that modify the configuration to adapt to different scenarios.

Configuring properties

If you need to modify the configuration, you can edit the .spec.properties field in the alluxio-cluster.yaml file.

The properties specified under the .spec.properties field will be appended to thealluxio-site.properties configuration file and the Alluxio processes will read that file. You can find your configurations in the Alluxio coordinator or worker pod by looking at /opt/alluxio/conf/alluxio-site.properties.

Setting the cache size

The size of the cache can be set by the .spec.worker.pagestore.size field. Note that by default the page store is located at the host path /mnt/alluxio/pagestore.

Besides the space for storing cached data, there is additional reserved space used by workers that must be accounted for when considering the total capacity of the storage device of the host. The size of the reserved space can be set by the .spec.worker.pagestore.reservedSize property and is recommended to be 5% - 10% of the cache size.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  worker:
    pagestore:
      size: 100Gi
      reservedSize: 10Gi

Change the resource limitations

For every component, like worker, coordinator, and FUSE, we can configure its resources by the following configuration:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  worker:
    count: 2
    resources:
      limits:
        cpu: "12"
        memory: "36Gi"
      requests:
        cpu: "1"
        memory: "32Gi"
    jvmOptions:
      - "-Xmx22g"
      - "-Xms22g"
      - "-XX:MaxDirectMemorySize=10g"
  coordinator:
    resources:
      limits:
        cpu: "12"
        memory: "36Gi"
      requests:
        cpu: "1"
        memory: "32Gi"
    jvmOptions:
      - "-Xmx4g"
      - "-Xms1g"

• The container will never be able to access the resource over the limits, and the requests are used during scheduling. For more information, please refer to Resource Management for Pods and Containers
• The total memory available should be a bit more than the sum of the heap size (-Xmx) and the direct memory size (-XX:MaxDirectMemorySize=10g) to avoid out-of-memory problems.

Use PVC for page store

To persist the cached data of a worker, specify a PVC for the worker's page store.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  worker:
    pagestore:
      type: persistentVolumeClaim
      storageClass: ""
      size: 100Gi
      reservedSize: 10Gi

• The PVC will be created by the operator

• The storageClass defaults to standard, but can be specified to empty string for static binding

• The size property specifies the size of the cache space. The reservedSize property specifies the amount of additional space used as an internal buffer for temporary data. The total size of the underlying storage will be the sum of the size of the cache and the reserved size. We recommend allocating a reserved size that's 5% - 10% of the size of the cache.

Mount customized config maps

A custom config map can be used to provide configuration files on pods. Although it can be used for other purposes such as environment variables, the following example will focus on files.

The following example will show how to mount a JSON file to specify the cache filter configuration, but the same steps can be applied for any file.

Create a new config map from a local file:

kubectl -n alx-ns create configmap cache-filter-cm --from-file=/path/to/cache_filter.json

Declare the config map with its mount point.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  configMaps:
    worker:
      cache-filter-cm: /opt/alluxio/conf/cachefilter/cache_filter.json
    coordinator:
      cache-filter-cm: /opt/alluxio/conf/cachefilter/cache_filter.json

The key is the name of the ConfigMap, and the value is the mounted path in the container.

Note that the /opt/alluxio/conf is already mounted by default, which prevents other files from being mounted directly within the conf/ directory. It is recommended to set the mounted path as a subdirectory of the conf/ directory.

Be sure to set the corresponding property value that defines the location of the configuration file. In the cache filter example, the following property needs to be set:alluxio.user.client.cache.filter.config.file: /opt/alluxio/conf/cachefilter/cache_filter.json

Add a file onto pods as a secret

This mechanism can be used to provide credentials files on pods.

Create a new secret from a local file:

kubectl -n alx-ns create secret my-file --from-file=/path/to/my-file

Specify which secrets to load and the file path on the pods.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  secrets:
    worker:
      my-file: /home/alluxio/my-file
    coordinator:
      my-file: /home/alluxio/my-file

Use the root user

The FUSE pod will always use the root user. The other processes use the user with uid 1000 by default. In the container, the user is named alluxio. To change it to the root user, use this configuration:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  user: 0
  group: 0
  fsGroup: 0

• Sometimes it’s enough to specify the .spec.fsGroup = 0 only when the files can be accessed by the root group

• The ownership of the mounted host path, such as the page store path and log path, will be transferred to root if changing to the root user.

Use external ETCD

If you have an external ETCD cluster, you can specify the endpoints for Alluxio to use.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    enabled: false
  properties:
    alluxio.etcd.endpoints: http://external-etcd:2379

When Client-to-server transport security with HTTPS, certificates are used for SSL/TLS connections to ETCD. For this, have a signed key pair (client.crt, pkcs8_key_encrypted.pem) and CA file (ca.crt) ready.

Here it needs a PKCS8 key, you can use the following command to convert the key:

$ openssl pkcs8 -topk8 -v2 aes256 -in server.key -out pkcs8_key_encrypted.pem

Note: If you use openssl pkcs8 -topk8 -nocrypt -in server.key -out pkcs8_key.pem generate unencrypted key file. You don’t need to set alluxio.etcd.tls.client.key.password in alluxio-site.properties.

Create secrets in Kubernetes with the created ca.crt, client.crt and pkcs8_key_encrypted.pem. For example,

$ kubectl -n alx-ns create secret generic etcd-certs --from-file=/path/to/ca.crt --from-file=/path/to/client.crt --from-file=/path/to/pkcs8_key_encrypted.pem

Configure the etcd properties in the alluxio-cluster.yaml file and specify the secrets for the coordinator, worker and fuse:

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  etcd:
    enabled: false
  properties:
    alluxio.etcd.endpoints: https://external-etcd:2379
    alluxio.etcd.tls.enabled: "true"
    alluxio.etcd.tls.ca.cert: /secrets/etcd-certs/ca.crt
    alluxio.etcd.tls.client.cert: /secrets/etcd-certs/client.crt
    alluxio.etcd.tls.client.key: /secrets/etcd-certs/pkcs8_key_encrypted.pem
    alluxio.etcd.tls.client.key.password: <your key password>
  secrets:
    coordinator:
      etcd-certs: /secrets/etcd-certs
    worker:
      etcd-certs: /secrets/etcd-certs
    fuse:
      etcd-certs: /secrets/etcd-certs

Deploy workers on nodes with different disk specs

The operator supports heterogeneous configurations for workers, specifically to configure different disk specs. Generally, inconsistencies within the worker configurations may lead to serious unexpected errors and therefore we do not support other scenarios other than the following use case.

1. Classify the nodes with the disk specs. e.g.: We have 10 nodes with one 1TB disk and 12 nodes with two 800GB disks.

2. Label the nodes to uniquely identify different groups of workers, where each group shares the same configuration.

# label nodes with one disk
kubectl label nodes <node name> apps.alluxio.com/disks=1
# label nodes with two disks
kubectl label nodes <node name> apps.alluxio.com/disks=2

1. Use .workerGroups to list the worker configurations, defining the labels to filter on with nodeSelector and its corresponding configuration.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
spec:
  # you can still specify common configurations with .worker
  worker:
    # the resources and the jvmOptions will affect all worker groups
    resources:
      limits:
        memory: 40Gi
      requests:
        memory: 36Gi
    jvmOptions: ["-Xmx20g", "-Xms20g", "-XX:MaxDirectMemorySize=16g"]
  # configuration here will override the one in worker
  workerGroups:
  - worker:
      count: 10
      nodeSelector:
        apps.alluxio.com/disks: 1
      pagestore:
        hostPath: /mnt/disk1/alluxio/pagestore
        size: 1Ti
  - worker:
      count: 12
      nodeSelector:
        apps.alluxio.com/disks: 2
      pagestore:
        hostPath: /mnt/disk1/alluxio/pagestore,/mnt/disk2/alluxio/pagestore
        size: 800Gi,800Gi