# Benchmarking S3 API Performance

COSBench (Cloud Object Storage Benchmark) is an open-source tool developed by Intel for stress testing object storage systems. Since Alluxio exposes an S3-compatible REST API, you can use COSBench to measure its read and write performance for S3-based workloads.

This guide explains how to set up and run COSBench to perform end-to-end performance tests against an Alluxio cluster.

## Performance Highlights

The following results were achieved using a 4-driver COSBench cluster testing a 4-worker Alluxio cluster. The tests measured read throughput against data fully cached in Alluxio.

### Large File Read Throughput (1GB files)

<table><thead><tr><th width="246.1796875">Concurrency per Driver</th><th>Total Throughput</th></tr></thead><tbody><tr><td>1 Thread</td><td>2.35 GB/s</td></tr><tr><td>16 Threads</td><td>20.44 GB/s</td></tr><tr><td>128 Threads</td><td>36.94 GB/s</td></tr></tbody></table>

### Small File Read IOPS (100KB files)

| Concurrency per Driver | Total Throughput | Total Operations/sec |
| ---------------------- | ---------------- | -------------------- |
| 1 Thread               | 50.26 MB/s       | 502 op/s             |
| 16 Threads             | 1.10 GB/s        | 11,302 op/s          |
| 128 Threads            | 4.69 GB/s        | 46,757 op/s          |

## Test Environment

The benchmark results were generated using the following environment, with all instances deployed in the same AWS availability zone.

* **COSBench Cluster**:
  * 1 Controller Node (`c5n.metal`)
  * 4 Driver Nodes (`c5n.metal`)
* **Alluxio Cluster**:
  * 1 Coordinator Node
  * 4 Worker Nodes (`i3en.metal` with 8 NVMe SSDs each)
* **Load Balancer**:
  * An AWS Elastic Load Balancer (ELB) was configured to distribute S3 requests evenly across the four Alluxio workers.

## Setup and Configuration

### Prerequisites

* **Operating System**: It is recommended to run COSBench on CentOS 7 or later.
* **Pre-loaded Data**: For read benchmarks, ensure your test dataset is fully loaded into the Alluxio cache from the underlying UFS.

### COSBench Installation

On all COSBench controller and driver nodes:

1. **Download COSBench**: Download [COSBench version 0.4.2.c4](https://github.com/intel-cloud/cosbench/releases/tag/v0.4.2.c4) and unzip it.
2. **Install Dependencies**:

   ```bash
   sudo yum install nmap-ncat curl java-1.8.0-openjdk-devel -y
   ```
3. **Disable MD5 Validation**: Edit the `cosbench-start.sh` script and add the following Java property to disable MD5 validation for S3 GET requests. This is necessary for compatibility with Alluxio's S3 API.

   ```properties
   -Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true
   ```
4. **Start COSBench Services**: From the COSBench root directory, start the controller and all drivers.

   ```bash
   sudo bash start-all.sh
   ```

## Running the Benchmark

### 1. Configure the Workload

Create an XML file (e.g., `s3-benchmark.xml`) to define the test workload. A COSBench workload consists of several stages:

* `init`: Creates the test buckets.
* `prepare`: Writes the initial data that will be used for testing.
* `main`: The primary testing stage. Runs the specified mix of read/write operations for a set duration.
* `cleanup`: Deletes the objects created during the test.
* `dispose`: Deletes the buckets.

**Note**: If you are using Alluxio mount points, you cannot create new buckets via the S3 API. You must skip the `init` and `dispose` stages and use pre-existing buckets that match your mount configuration.

### Example: Basic Read/Write Workload

This example shows a simple workload that creates two buckets, writes 10 objects of 64KB to each, runs a 30-second test with an 80/20 read-write ratio, and then cleans up.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-sample" description="sample benchmark for s3">
  <storage type="s3" config="accesskey=root;secretkey=dump;endpoint=http://localhost:29998;path_style_access=true" />

  <workflow>
    <workstage name="init">
      <work type="init" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
    </workstage>

    <workstage name="prepare">
      <work type="prepare" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,10);sizes=c(64)KB" />
    </workstage>

    <workstage name="main">
      <work name="main" workers="8" runtime="30">
        <operation type="read" ratio="80" config="cprefix=s3testqwer;containers=u(1,2);objects=u(1,10)" />
        <operation type="write" ratio="20" config="cprefix=s3testqwer;containers=u(1,2);objects=u(11,20);sizes=c(64)KB" />
      </work>
    </workstage>

    <workstage name="cleanup">
      <work type="cleanup" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,20)" />
    </workstage>

    <workstage name="dispose">
      <work type="dispose" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
    </workstage>
  </workflow>
</workload>
```

### Example: High-Concurrency Read Test

This example prepares 10,000 small objects (100KB) and then uses four drivers, each with 128 worker threads, to read the objects concurrently for 300 seconds.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-sample" description="sample benchmark for s3">
  <storage type="s3" config="accesskey=root;secretkey=dump;endpoint=http://<ip>:29998;path_style_access=true" />
  <workflow>
    <workstage name="prepare">
      <work type="prepare" workers="100" config="cprefix=ufs;containers=r(2,2);oprefix=myobjects;osuffix=.jpg;objects=r(1,10000);sizes=c(100)KB" />
    </workstage>

    <workstage name="128">
      <work name="read" workers="128" driver="driver1" runtime="300">
        <operation type="read" ratio="100" config="cprefix=ufs;containers=r(2,2);oprefix=myobjects;osuffix=.jpg;objects=u(1,10000)" />
      </work>
      <work name="read" workers="128" driver="driver2" runtime="300">
        <operation type="read" ratio="100" config="cprefix=ufs;containers=r(2,2);oprefix=myobjects;osuffix=.jpg;objects=u(1,10000)" />
      </work>
      <work name="read" workers="128" driver="driver3" runtime="300">
        <operation type="read" ratio="100" config="cprefix=ufs;containers=r(2,2);oprefix=myobjects;osuffix=.jpg;objects=u(1,10000)" />
      </work>
      <work name="read" workers="128" driver="driver4" runtime="300">
        <operation type="read" ratio="100" config="cprefix=ufs;containers=r(2,2);oprefix=myobjects;osuffix=.jpg;objects=u(1,10000)" />
      </work>
    </workstage>
  </workflow>
</workload>
```

### 2. Submit the Workload

Use the `cli.sh` script to submit your workload XML file to the COSBench controller.

```bash
bash cli.sh submit conf/s3-benchmark.xml
```

### 3. Monitor the Results

You can monitor the status and view the results of your benchmark jobs from the COSBench web interface, available at `http://<CONTROLLER_IP>:19088/controller/index.html`.

### 4. Stop COSBench Services

Once you have completed your tests, stop the services.

```bash
sudo bash stop-all.sh
```