Tencent COS
Last updated
Last updated
This guide describes how to configure Alluxio with Tencent (Cloud Object Storage) as the under storage system. Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for massive data and accessible via HTTP/HTTPS protocols. It can store massive amounts of data and features imperceptible bandwidth and capacity expansion, making it a perfect data pool for big data computation and analytics.
Alluxio runs on multiple machines in cluster mode so its binary package needs to be deployed on the machines.
Before using COS with Alluxio, either create a new bucket or use an existing one. Additionally, identify the directory you wish to use within that bucket, whether by creating a new directory or selecting an existing one. For this guide, the COS bucket name is COS_BUCKET, the directory within the bucket is COS_DATA, and the bucket region is COS_REGION.
Use the to add a new mount point, specifying the Alluxio path to create the mount on and the COS path as the UFS URI. Credentials and configuration options can also be specified as part of the mount command by specifying the --option flag as described by .
An example command to mount cos://<COS_ALLUXIO_BUCKET>/<COS_DATA> to /cos:
Note that if you want to mount the root of the COS bucket, add a trailing slash after the bucket name (e.g. cos://COS_BUCKET/).
Note that configuration options can be specified as mount options or as configuration properties in conf/alluxio-site.properties. The following sections will describe how to set configurations as properties, but they can also be set as mount options via --option <key>=<value>.
To enable the use of the HTTPS protocol for secure communication with COS with an additional layer of security for data transfers, configure the following setting in conf/alluxio-site.properties:
The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading.
To enable COS multipart upload, you need to modify conf/alluxio-site.properties to include:
There are other parameters you can specify in conf/alluxio-site.properties to potentially speed up the upload.
Sometimes there may be an error in accessing UFS because the server is temporarily unable to respond. You can configure a retry policy for UFS requests.
Each I/O request sent to UnderFS like getObject, putObject, MultipartUpload, Alluxio will check the response. If the response is an error, and the error code suggests it may be retryable, the request will be resubmitted according to the retry policy in configuration. Alluxio will keep trying until the request is successful or reaches the maximum number of retries. The wait interval between successive retries will gradually increase from the configured base sleep time to the maximum sleep time.
The following error codes are categorized as retryable: 500 HTTP_INTERNAL_ERROR, 502 HTTP_BAD_GATEWAY, 503 HTTP_UNAVAILABLE, 503 Slow Down, and 504 HTTP_GATEWAY_TIMEOUT.
Note:
4xx status code usually represents client errors, such as NOT_FOUND, PERMISSION_DENIED, UNAUTHENTICATED, etc. Such errors should never be retried since the issue is on the client side.
5xx status code usually represents server errors, but not all 5xx error should be retried. For example, 501 HTTP_NOT_IMPLEMENTED should not be retried.
If you want to set the retry policy for the UFS accessing request, you need to modify conf/alluxio-site.properties to include: