Alibaba Cloud OSS

This guide describes how to configure Alibaba Cloud OSS as Alluxio's under storage system.

Alibaba Cloud Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Alibaba Cloud. OSS provides multiple storage classes to help you manage and reduce storage costs.

For more information about Alibaba Cloud OSS, please read its documentation

Prerequisites

Before you get started, please ensure you have the required information listed below:

<OSS_BUCKET>

Create a new bucket in the OSS console or use an existing bucket

<OSS_DIRECTORY>

The directory you want to use in the bucket, either by creating a new directory or using an existing one

<OSS_ACCESS_KEY_ID>

ID used to identify a user. See How to Obtain AccessKey Pair

<OSS_ACCESS_KEY_SECRET>

Secret used to verify the identity of the user. See How to Obtain AccessKey Pair

<OSS_ENDPOINT>

Endpoints are the domain names that other services can use to access OSS. See Regions and OSS Endpoints in the Public Cloud

Basic Setup

For the general mount mechanism and UnderFileSystem CR field reference, see Underlying Storage.

An example ufs.yaml to create an OSS mount point with the operator:

apiVersion: k8s-operator.alluxio.com/v1
kind: UnderFileSystem
metadata:
  name: alluxio-oss
  namespace: alx-ns
spec:
  alluxioCluster: alluxio-cluster
  path: oss://<OSS_BUCKET>/<OSS_DIRECTORY>
  mountPath: /oss
  mountOptions:
    fs.oss.accessKeyId: <OSS_ACCESS_KEY>
    fs.oss.accessKeySecret: <OSS_ACCESS_KEY_SECRET>
    fs.oss.endpoint: <OSS_ENDPOINT>

An example command to mount oss://<OSS_BUCKET>/<OSS_DIRECTORY> to /oss if not using the operator:

bin/alluxio mount add --path /oss/ --ufs-uri oss://<OSS_BUCKET>/<OSS_DIRECTORY> \
  --option fs.oss.accessKeyId=<OSS_ACCESS_KEY> --option fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET> \
  --option fs.oss.endpoint=<OSS_ENDPOINT>

Note that if you want to mount the root of the OSS bucket, add a trailing slash after the bucket name (e.g. oss://OSS_BUCKET/).

Advanced Setup

Note that configuration options can be specified as mount options or as configuration properties in conf/alluxio-site.properties. The following sections will describe how to set configurations as properties, but they can also be set as mount options via --option <key>=<value>.

Enabling HTTPS

To enable the use of the HTTPS protocol for secure communication with OSS with an additional layer of security for data transfers, configure the following setting in conf/alluxio-site.properties:

alluxio.underfs.oss.secure.http.enabled=true

OSS multipart upload

We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading.

There are other parameters you can specify in conf/alluxio-site.properties to potentially speed up the upload.

# Timeout for uploading part when using multipart upload.
alluxio.underfs.object.store.multipart.upload.timeout

# Thread pool size for OSS multipart upload.
alluxio.underfs.oss.multipart.upload.threads

# Multipart upload partition size for OSS. The default partition size is 16MB.
alluxio.underfs.oss.multipart.upload.part.size

Disable OSS multipart upload, the upload method uploads one file completely from start to end in one go. you need to modify conf/alluxio-site.properties to include:

alluxio.underfs.oss.multipart.upload.enabled=false

Setting Request Retry Policy

For retry configuration that applies to all object store UFS types, see Request Retry Policy in the UFS overview.

Last updated 2 months ago