Azure Blob Storage

This guide describes how to configure Alluxio with Azure Blob Store as the under storage system.

Prerequisites

In preparation for using Azure Blob Store with Alluxio, create a new container in your Azure storage account or use an existing container. You should also note that the directory you want to use in that container, either by creating a new directory in the container, or using an existing one. For the purposes of this guide, the Azure storage account name is called <AZURE_ACCOUNT>, the container in that storage account is called <AZURE_CONTAINER> and the directory in that container is called <AZURE_DIRECTORY>. For more information about Azure storage account, Please seehere.

Basic Setup

For the general mount mechanism and UnderFileSystem CR field reference, see Underlying Storage.

Mount the container through the abfs scheme, which is served by the ABFS driver. The container is part of the authority, separated from the account host by @:

abfs://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.dfs.core.windows.net/<AZURE_DIRECTORY>/

Kubernetes (Operator)

An example ufs.yaml to create an Azure Blob Storage mount point with the operator:

apiVersion: k8s-operator.alluxio.com/v1
kind: UnderFileSystem
metadata:
  name: alluxio-abfs
  namespace: alx-ns
spec:
  alluxioCluster: alluxio-cluster
  path: abfs://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.dfs.core.windows.net/<AZURE_DIRECTORY>/
  mountPath: /abfs
  mountOptions:
    fs.azure.account.key.<AZURE_ACCOUNT>.dfs.core.windows.net: <YOUR ACCESS KEY>

Docker / Bare-Metal

An example command to mount abfs://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.dfs.core.windows.net/<AZURE_DIRECTORY>/ to /abfs if not using the operator:

bin/alluxio mount add --path /abfs/ --ufs-uri abfs://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.dfs.core.windows.net/<AZURE_DIRECTORY>/ \
  --option fs.azure.account.key.<AZURE_ACCOUNT>.dfs.core.windows.net=<YOUR ACCESS KEY>

This property key fs.azure.account.key.<AZURE_ACCOUNT>.dfs.core.windows.net will let the Alluxio use the AccountKey to access Azure Blob Store.

Both abfs:// and abfss:// reach this driver. abfss:// always uses HTTPS; abfs:// does too unless Hadoop's fs.azure.always.use.https is turned off. Either works for a mount.

The endpoint in the key name has to be dfs, matching the URI. Alluxio recognizes an account key only under fs.azure.account.key.<AZURE_ACCOUNT>.dfs.core.windows.net; one named for the blob endpoint is not seen as a key at all, and the mount falls back to managed identity, which then fails with a token error rather than anything naming the credential.

WASB Mounts

WASB is no longer the recommended way to reach Azure Storage. The com.microsoft.azure.storage client it depends on belongs to the Azure Storage Java Track 1 SDK, which Microsoft retired on 13 September 2024. Existing WASB mounts may keep working, but create new mounts with abfs://.

The wasb:// and wasbs:// schemes are served by the WASB driver, which reaches the account over its blob endpoint and takes the account key under a different property name:

wasbs://<AZURE_CONTAINER>@<AZURE_ACCOUNT>.blob.core.windows.net/<AZURE_DIRECTORY>/
fs.azure.account.key.<AZURE_ACCOUNT>.blob.core.windows.net=<YOUR ACCESS KEY>

If a mount uses wasb:// rather than wasbs://, an account that requires HTTPS reports:

alluxio.exception.AlluxioException: com.microsoft.azure.storage.StorageException: The account being accessed does not support http.

Change the path to start with wasbs://.