Alluxio
ProductsLanguageHome
Al-3.1
Al-3.1
  • Overview
  • Get Started
  • Storage Integrations
    • Storage Integrations Overview
    • Amazon AWS S3
    • HDFS
    • Aliyun OSS
    • COS
  • Operations
  • Reference
Powered by GitBook
On this page
  • Prerequisites
  • Basic Setup
  • Advanced Setup
  • [Experimental] OSS multipart upload
  1. Storage Integrations

Aliyun OSS

Last updated 7 months ago

This guide describes how to configure as Alluxio's under storage system.

Aliyun Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Alibaba Cloud. OSS provides multiple storage classes to help you manage and reduce storage costs.

For more information about Aliyun OSS, please read its

Prerequisites

If you haven't already, please see before you get started.

In preparation for using OSS with Alluxio:

<OSS_BUCKET>

<OSS_DIRECTORY>

The directory you want to use in the bucket, either by creating a new directory or using an existing one

<OSS_ACCESS_KEY_ID>

<OSS_ACCESS_KEY_SECRET>

<OSS_ENDPOINT>

Basic Setup

To use Aliyun OSS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Specify an OSS bucket and directory as the underfs address by modifying conf/alluxio-site.properties.

For example, the under storage address can be oss://alluxio-bucket/ if you want to mount the whole bucket to Alluxio, or oss://alluxio-bucket/alluxio/data if only the directory /alluxio/data inside the oss bucket alluxio-bucket is mapped to Alluxio.

alluxio.dora.client.ufs.root=oss://<OSS_BUCKET>/<OSS_DIRECTORY>

Specify credentials for Aliyun OSS access by adding the following properties in conf/alluxio-site.properties:

fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID>
fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET>
fs.oss.endpoint=<OSS_ENDPOINT>

Advanced Setup

[Experimental] OSS multipart upload

The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading.

To enable OSS multipart upload, you need to modify conf/alluxio-site.properties to include:

alluxio.underfs.oss.multipart.upload.enabled=true

There are other parameters you can specify in conf/alluxio-site.properties to make the process faster and better.

# Timeout for uploading part when using multipart upload.
alluxio.underfs.object.store.multipart.upload.timeout
# Thread pool size for OSS multipart upload.
alluxio.underfs.oss.multipart.upload.threads
# Multipart upload partition size for OSS. The default partition size is 64MB. 
alluxio.underfs.oss.multipart.upload.partition.size

or use an existing bucket

ID used to identify a user. See

Secret used to verify the identity of the user. See

Endpoints are the domain names that other services can use to access OSS. See

Create a new bucket in the OSS console
How to Obtain AccessKey Pair
How to Obtain AccessKey Pair
Regions and OSS Endpoints in the Public Cloud
Aliyun OSS
documentation
Prerequisites