Handling Images

Two sets of container images are required for deployment:

1. Alluxio images, provided by your Alluxio sales representative

2. Publicly accessible images for third party components

The images must be served by an image registry accessible to the Kubernetes cluster. For the Alluxio images, it is expected that they are uploaded to a registry managed by the user.

An image registry is a centralized location for storing and sharing your container images. It can be either public or private. Cloud provider may come with its container registry as a service. For example, Amazon Elastic Container Registry(ECR), Azure Container Registry (ACR), and Google Container Registry (GCR). Private registries may also be provided from your local system or within private networks of your organization.

Alluxio images

The following image files are provided:

• alluxio-operator-3.2.1-docker.tar is the docker image for all Alluxio operator components

• alluxio-enterprise-AI-3.6-12.0.2-docker.tar is the docker image for Alluxio coordinator and workers

Optionally, the following images may also be provided:

• alluxio-gateway-AI-3.6-12.0.2-docker.tar is the docker image for Alluxio's API gateway

• alluxio-dashboard-AI-3.6-12.0.2.tar is the docker image for Alluxio's Management Console

This example shows how to upload the Alluxio operator images.

# load the image to local
$ docker load -i alluxio-operator-3.2.1-docker.tar
$ docker load -i alluxio-enterprise-AI-3.6-12.0.2-docker.tar

# retag the image with your private registry
$ docker tag alluxio/operator:3.2.1 <PRIVATE_REGISTRY>/alluxio-operator:3.2.1
$ docker tag alluxio/alluxio-enterprise:AI-3.6-12.0.2 <PRIVATE_REGISTRY>/alluxio-enterprise:AI-3.6-12.0.2

# push to the remote registry
$ docker push <PRIVATE_REGISTRY>/alluxio-operator:3.2.1
$ docker push <PRIVATE_REGISTRY>/alluxio-enterprise:AI-3.6-12.0.2

Unable to access public image registry

In the case that the Kubernetes cluster does not have access to the public internet, the images usually supplied by public image registries will not be accessible to download. These image must also be uploaded to an image registry accessible to the Kubernetes cluster to successfully deploy Alluxio. If your network environment cannot access the public image registry, you will encounter a timeout error when pulling the images:

# Check if the operator is running properly
$ kubectl -n alluxio-operator get pod
NAME                                              READY   STATUS              RESTARTS   AGE
alluxio-cluster-controller-65b59f65b4-5d667       1/1     Running             0          22s
alluxio-collectinfo-controller-667b746fd6-hfzqk   1/1     Running             0          22s
alluxio-csi-controller-c85f8f759-sqc56            0/2     ContainerCreating   0          22s
alluxio-csi-nodeplugin-5pgmg                      0/2     ContainerCreating   0          22s
alluxio-csi-nodeplugin-fpkcq                      0/2     ContainerCreating   0          22s
alluxio-csi-nodeplugin-j9wll                      0/2     ContainerCreating   0          22s
alluxio-ufs-controller-5f69bbb878-7km58           1/1     Running             0          22s

You may notice that the cluster controller, ufs controller and collectinfo controller have started successfully, but the csi controller and csi nodeplugin remain in the ContainerCreating state. This is due to a timeout while pulling the dependent images. By using kubectl describe pod to view detailed information, you will see error messages similar to the following:

$ kubectl -n alluxio-operator describe pod -l app.kubernetes.io/component=csi-controller

Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       10m                    default-scheduler  Successfully assigned alluxio-operator/alluxio-csi-controller-c85f8f759-sqc56 to <nodeName>
  Normal   AllocIPSucceed  10m                    terway-daemon      Alloc IP 10.0.0.27/24 took 28.443992ms
  Normal   Pulling         10m                    kubelet            Pulling image "registry.xxx.com/alluxio/operator:3.2.1"
  Normal   Pulled          10m                    kubelet            Successfully pulled image "registry.xxx.com/alluxio/operator:3.2.1" in 5.55s (5.55s including waiting)
  Normal   Created         10m                    kubelet            Created container csi-controller
  Normal   Started         10m                    kubelet            Started container csi-controller
  Warning  Failed          8m20s (x2 over 10m)    kubelet            Failed to pull image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": failed to pull and unpack image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": failed to resolve reference "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/sig-storage/csi-provisioner/manifests/v2.0.5": dial tcp 142.251.8.82:443: i/o timeout
  Warning  Failed          8m20s (x3 over 10m)    kubelet            Error: ErrImagePull
  Warning  Failed          7m40s (x5 over 10m)    kubelet            Error: ImagePullBackOff
  Warning  Failed          6m56s (x2 over 9m19s)  kubelet            Failed to pull image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": failed to resolve reference "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/sig-storage/csi-provisioner/manifests/v2.0.5": dial tcp 64.233.187.82:443: i/o timeout
  Normal   Pulling         5m29s (x5 over 10m)    kubelet            Pulling image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5"
  Normal   BackOff         30s (x28 over 10m)     kubelet            Back-off pulling image "registry.k8s.io/sig-storage/csi-provisioner:v2.0.5"

Third party dependent images

Component	Image Name	Version	Purpose
operator CSI	registry.k8s.io/sig-storage/csi-node-driver-registrar	v2.0.0	csi driver registrar dependency
operator CSI	registry.k8s.io/sig-storage/csi-provisioner	v2.0.5	csi provisioner dependency
cluster ETCD	docker.io/bitnami/etcd	3.5.9-debian-11-r24	etcd dependency
cluster ETCD	docker.io/bitnami/os-shell	11-debian-11-r2	os-shell dependency
cluster monitor	grafana/grafana	10.4.5	Monitoring dashboard
cluster monitor	prom/prometheus	v2.52.0	Metrics collection

The commands to pull the Docker images and upload them to your private image registry are as follows; be sure to set platform (ex. linux/amd64) that matches the Kubernetes cluster environment.

# Pull the Docker images
$ docker pull --platform linux/amd64 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.0.0
$ docker pull --platform linux/amd64 registry.k8s.io/sig-storage/csi-provisioner:v2.0.5
$ docker pull --platform linux/amd64 docker.io/bitnami/etcd:3.5.9-debian-11-r24
$ docker pull --platform linux/amd64 docker.io/bitnami/os-shell:11-debian-11-r2
$ docker pull --platform linux/amd64 grafana/grafana:10.4.5
$ docker pull --platform linux/amd64 prom/prometheus:v2.52.0

# Tag the images with your private registry
$ docker tag registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.0.0 <PRIVATE_REGISTRY>/csi-node-driver-registrar:v2.0.0
$ docker tag registry.k8s.io/sig-storage/csi-provisioner:v2.0.5 <PRIVATE_REGISTRY>/csi-provisioner:v2.0.5
$ docker tag docker.io/bitnami/etcd:3.5.9-debian-11-r24 <PRIVATE_REGISTRY>/etcd:3.5.9-debian-11-r24
$ docker tag docker.io/bitnami/os-shell:11-debian-11-r2 <PRIVATE_REGISTRY>/os-shell:11-debian-11-r2
$ docker tag grafana/grafana:10.4.5 <PRIVATE_REGISTRY>/grafana:10.4.5
$ docker tag prom/prometheus:v2.52.0 <PRIVATE_REGISTRY>/prometheus:v2.52.0

# Push the images to your private registry
$ docker push <PRIVATE_REGISTRY>/csi-node-driver-registrar:v2.0.0
$ docker push <PRIVATE_REGISTRY>/csi-provisioner:v2.0.5
$ docker push <PRIVATE_REGISTRY>/etcd:3.5.9-debian-11-r24
$ docker push <PRIVATE_REGISTRY>/os-shell:11-debian-11-r2
$ docker push <PRIVATE_REGISTRY>/grafana:10.4.5
$ docker push <PRIVATE_REGISTRY>/prometheus:v2.52.0

Updating alluxio-operator.yaml

Update the image addresses in the alluxio-operator/alluxio-operator.yaml file, adding the provisioner and driverRegistrar image addresses:

global:
  image: <PRIVATE_REGISTRY>/alluxio-operator
  imageTag: 3.2.1

alluxio-csi:
  controllerPlugin: 
    provisioner: 
      image: <PRIVATE_REGISTRY>/csi-provisioner:v2.0.5
  nodePlugin: 
    driverRegistrar: 
        image: <PRIVATE_REGISTRY>/csi-node-driver-registrar:v2.0.0

Updating alluxio-cluster.yaml

Modify the image addresses in the alluxio-operator/alluxio-cluster.yaml file accordingly.

apiVersion: k8s-operator.alluxio.com/v1
kind: AlluxioCluster
metadata:
  name: alluxio-cluster
  namespace: alx-ns
spec:
  image: <PRIVATE_REGISTRY>/alluxio-enterprise
  imageTag: AI-3.6-12.0.2
  properties:
  worker:
    count: 2
    pagestore:
      size: 100Gi
  etcd:
    image:
      registry: <PRIVATE_REGISTRY>
      repository: <PRIVATE_REPOSITORY>/etcd
      tag: 3.5.9-debian-11-r24
    volumePermissions:
      image:
        registry: <PRIVATE_REGISTRY>
        repository: <PRIVATE_REPOSITORY>/os-shell
        tag: 11-debian-11-r2
  prometheus:
    image: <PRIVATE_REGISTRY>/prometheus
    imageTag: v2.52.0
  grafana:
    image: <PRIVATE_REGISTRY>/grafana
    imageTag: 10.4.5