Alluxio
ProductsLanguageHome
  • Introduction
  • Overview
    • Architecture
    • Job Service
    • Quick Start Guide
    • FAQ
    • Use Cases
  • Core Services
    • Caching
    • Unified Namespace
  • Install Alluxio
    • Local Machine
    • Cluster
    • Cluster with HA
    • Docker
    • Software Requirements
  • Kubernetes
    • Deploy
    • Spark on Kubernetes
    • Metrics
  • Cloud Native
    • Alibaba Cloud ACK
    • AWS EMR
    • Tencent EMR
    • Google Dataproc
  • Compute Integration
    • Apache Spark
    • Apache Hadoop MapReduce
    • Apache Flink
    • Apache Hive
    • Presto on Iceberg (Experimental)
    • Presto
    • Trino
    • Tensorflow
  • Storage Integrations
    • Amazon AWS S3
    • HDFS
    • Azure Blob Store
    • Azure Data Lake Storage Gen2
    • Azure Data Lake Storage
    • Google Cloud Storage
    • Qiniu Kodo
    • COSN
    • CephObjectStorage
    • MinIO
    • NFS
    • Aliyun Object Storage Service
    • Ozone
    • Swift
    • WEB
    • CephFS
  • Security
  • Operations
    • Configuration Settings
    • User CLI
    • Admin CLI
    • Web UI
    • Journal Management
    • Metastore Management
    • Metrics
  • Administration
    • Troubleshooting
    • Basic Logging
    • Remote Logging
    • Performance Tuning
    • Scalability Tuning
    • StressBench (Experimental)
    • Upgrading
  • Solutions
  • Client APIs
    • Java API
    • S3 API
    • REST API
    • POSIX API
  • Contributor Resources
    • Building Alluxio From Source
    • Contribution Guide
    • Code Conventions
    • Documentation Conventions
    • Contributor Tools
  • Reference
    • List Of Configuration Properties
    • List of Metrics
  • REST API
    • Master REST API
    • Worker REST API
    • Proxy REST API
    • Job REST API
  • Javadoc
Powered by GitBook
On this page
  • Prerequisites
  • Basic Setup
  • Example: Running Alluxio Locally with OSS
  • Advanced Setup
  • Nested Mount
  1. Storage Integrations

Aliyun Object Storage Service

Last updated 6 months ago

This guide describes how to configure as Alluxio's under storage system. Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Aliyun.

Prerequisites

The Alluxio binaries must be on your machine. You can either , or .

In preparation for using OSS with Alluxio, follow the to sign up for OSS and create an OSS bucket.

Basic Setup

To configure Alluxio to use OSS as under storage, you will need to modify the configuration file conf/alluxio-site.properties. If the file does not exist, create the configuration file from the template.

$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Edit conf/alluxio-site.properties file to set the under storage address to the OSS bucket and the OSS directory you want to mount to Alluxio. For example, the under storage address can be oss://alluxio-bucket/ if you want to mount the whole bucket to Alluxio, or oss://alluxio-bucket/alluxio/data if only the directory /alluxio/data inside the oss bucket alluxio-bucket is mapped to Alluxio.

alluxio.master.mount.table.root.ufs=oss://<OSS_BUCKET>/<OSS_DIRECTORY>

Specify the Aliyun credentials for OSS access. In conf/alluxio-site.properties, add:

fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID>
fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET>
fs.oss.endpoint=<OSS_ENDPOINT>

fs.oss.accessKeyId and fs.oss.accessKeySecret is the for OSS, which are created and managed in .

fs.oss.endpoint is the internet endpoint of this bucket, which can be found in the bucket overview page with values like oss-us-west-1.aliyuncs.com and oss-cn-shanghai.aliyuncs.com. Available endpoints are listed in the .

Example: Running Alluxio Locally with OSS

Start the Alluxio servers:

$ ./bin/alluxio format
$ ./bin/alluxio-start.sh local

Run a simple example program:

$ ./bin/alluxio runTests

Visit your OSS directory oss://<OSS_BUCKET>/<OSS_DIRECTORY> to verify the files and directories created by Alluxio exist. For this test, you should see files named like <OSS_BUCKET>/<OSS_DIRECTORY>/default_tests_files/BasicFile_CACHE_PROMOTE_MUST_CACHE.

Stop Alluxio by running:

$ ./bin/alluxio-stop.sh local

Advanced Setup

Nested Mount

$ ./bin/alluxio fs mount --option fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID> \
  --option fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET> \
  --option fs.oss.endpoint=<OSS_ENDPOINT> \
  /oss oss://<OSS_BUCKET>/<OSS_DIRECTORY>/

This will start an Alluxio master and an Alluxio worker. You can see the master UI at .

An OSS location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio's can be used for this purpose. For example, the following command mounts a directory inside an OSS bucket into Alluxio directory /oss:

Aliyun OSS
compile Alluxio
download the binaries locally
OSS quick start guide
AccessKey
Aliyun AccessKey management console
OSS Internet Endpoints documentation
http://localhost:19999
Mount Command