COSN
This guide describes how to configure Alluxio with Tencent COS (Cloud Object Storage) as the under storage system. Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data and accessible via HTTP/HTTPS protocols. It can store massive amounts of data and features imperceptible bandwidth and capacity expansion, making it a perfect data pool for big data computation and analytics.
Basic Setup
Alluxio runs on multiple machines in cluster mode so its binary package needs to be deployed on the machines. You can either compile Alluxio or download the binaries locally.
In preparation for using COS with Alluxio, create a new bucket or use an existing bucket. You should also note the directory you want to use in that bucket, either by creating a new directory in the bucket or using an existing one. For the purposes of this guide, the COS Bucket name is called COSN_ALLUXIO_BUCKET
, the directory in that bucket is called COSN_DATA
, and COS Bucket region is called COSN_REGION
which specifies the region of your bucket.
Basic Setup
Alluxio unifies access to different storage systems through the unified namespace feature. COSN UFS is used to access Tencent Cloud object storage and a COS location can be either mounted at the root of the Alluxio namespace or as a nested directory.
Root Mount Point
Create conf/alluxio-site.properties
and conf/core-site.xml
if they do not exist.
Configure Alluxio to use COSN as its under storage system by modifying conf/alluxio-site.properties
and conf/core-site.xml
. Specify an existing COS bucket and directory as the under storage system by modifying conf/alluxio-site.properties
to include:
Specify COS configuration information in order to access COS by modifying conf/core-site.xml
to include:
The above is the most basic configuration. For more configuration please refer to here. After these changes, Alluxio should be configured to work with COSN as its under storage system and you can try Running Alluxio Locally with COSN.
Nested Mount
A COS location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. The mount command can be used for this purpose.
Running Alluxio Locally with COSN
Start up Alluxio locally to see that everything works.
This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.
Run a simple example program:
Visit your COS directory at COSN_ALLUXIO_BUCKET/COSN_DATA
to verify the files and directories created by Alluxio exist. For this test, you should see files named like:
To stop Alluxio, you can run:
Last updated