COSBench performance benchmark
COSBench Overview
COSBench is an open-source stress testing tool developed by Intel for object storage.
Alluxio supports a RESTful API that is compatible with the basic operations of the Amazon S3 API. this tool can be used for read-write performance testing.
This document describes how to conduct end-to-end testing of Alluxio using COSBench.
Results Summary
File size | File count | Concurrency of each driver | Throughput |
---|---|---|---|
1GB | 10000 | 1 | driver1: 604.03 MB/S driver2: 592.85 MB/S driver3: 597.94 MB/S driver4: 612.99 MB/S total: 2407.81 MB/S |
1GB | 10000 | 16 | driver1: 5.45 GB/S driver2: 5.1 GB/S driver3: 4.69 GB/S driver4: 5.2 GB/S total: 20.44 GB/s |
1GB | 10000 | 128 | driver1: 9.24 GB/S driver2: 8.59 GB/S driver3: 10.06 GB/S driver4: 9.05 GB/S total: 36.94 GB/s |
100KB | 10000 | 1 | driver1: 14.61 MB/S - 146.08 op/s driver2: 11.96 MB/S - 119.55 op/s driver3: 11.75 MB/S - 117.46 op/s driver4: 11.94 MB/S - 119.42 op/s total: 50.26 MB/S - 502.51 op/s |
100KB | 10000 | 16 | driver1: 311.69 MB/S - 3116.87 op/s driver2: 244.72 MB/S - 2447.15 op/s driver3: 286.34 MB/S - 2863.35 op/s driver4: 287.53 MB/S - 2875.26 op/s total: 1130.28 MB/S - 11302.63 op/s |
100KB | 10000 | 128 | driver1: 1.19 GB/S - 11877.99 op/s driver2: 1.17 GB/S - 11660.19 op/s driver3: 1.16 GB/S - 11569.5 op/s driver4: 1.17 GB/S - 11650.17 op/s total: 4.69 GB/S - 46757.85 op/s |
The test results are based on an Alluxio cluster configured as follows, with all server instances available on AWS:
COSBench Cluster: 1 COSBench Controller node and 4 COSBench Driver nodes.
COSBench Instance: c5n.metal: 72 core + 192GB memory + 100Gb network
Alluxio Cluster: 1 Alluxio Coordinator node and 4 Alluxio Worker nodes.
Alluxio Worker Instance: i3en.metal: 96 core + 768GB memory + 100Gb network + 8 nvme
AWS ELB load balancer: Elastic Load Balancing
Preparing the Test Environment
The tool is recommended to run on CentOS 7.0 and above, as there may be unexpected issues in the Ubuntu environment.
Configuring Alluxio
For other Alluxio-related configurations, refer to the Fio Tests section.
You can configure one or more Alluxio Workers as a cache cluster.
Need to configure a load balancer to evenly distribute requests to all Alluxio Worker nodes.
Ensure that the dataset has been completely loaded into the Alluxio cache from UFS.
Preparing COSBench tool
Download the COSBench 0.4.2.c4.zip file from the COSBench GitHub website, and decompress it on the server.
Execute the following command to install the dependency library of COSBench.
For CentOS, execute the following command to install dependencies:
Edit the
cosbench-start.sh
file, add the following parameters to the jvm properties, and disable the md5 verification function of s3:
Start COSBench controller and driver:
For Centos, execute the following command:
Running the test
Edit the s3-config-sample.xml file and add task configuration information. Task configuration mainly includes the following five stages:
init stage: Create the bucket based on the prefix, suffix, and number of containers configured.
prepare stage: Write the data to the bucket for the subsequent testing, specify threads count, file size and file count according to the configuration.
main stage: The main stage of testing can specify the running time and the number of threads to perform the corresponding read and write operations.
cleanup stage: This stage is to clean up the environment, mainly by deleting the data in the bucket to ensure that the data will not be retained in the cluster.
dispose stage: delete the bucket.
In the test, all stages can be executed or only some stages can be executed according to actual needs.
Note: When using the mount table operation, it can not create a new bucket. You should skip the init and dispose stages and manually create the mount point that meet the bucket name prefix and container naming rules for subsequent testing.
Parameter description:
Parameter Description accesskey、secretkey
Currently, the Alluxio S3 API does not support access keys and secret keys. When supplying an access key to the test tool, put the intended Alluxio ACL username. The secret key is unused, so you may use any dummy value.
cprefix
Bucket name prefix, such as "examplebucket"
containers
For the range of bucket name values, the final bucket name consists of cprefix, containers, and csuffix (if this parameter is set), such as "examplebucket1test", "examplebucket2test"
csuffix
Optional, bucket name suffix, such as "test"
oprefix
Object name prefix, such as "testobject", "dir/testobject"
objects
The number and range of objects, such as "r(1,10000)", "r(200,500)"
osuffix
Object name suffix, such as ".jpg", ".csv"
radio
The read-write ratio of the current task
runtime
test running time
workers
The number of threads for the current task
The example configuration is as follows:
Run the task:
Check the task result
Check the task status and result in the endpoint "http://ip:19088/controller/index.html" (Replace IP with the IP of the COSBench controller node):
The following is a read performance test for high concurrency reading of 10k 100KB files, including the following two stages:
prepare stage: 100 worker threads upload 10k 100KB objects.
main stage: 4 drivers and 128 worker threads each driver read objects concurrently, running for 300 seconds
After conducting performance pressure tests in Stage 1 and Stage 2, the results are as follows:
Execute the following command to stop the testing service
Last updated