REST API

The Rest APIs are designed for jobs, configurations, and cluster management.

Distributed Load

Pre-loads UFS (Underlying File System) data into worker caches to prevent performance degradation caused by cold reads.

Submit or resume a load job

Method

Path

Parameters

Request Body Type

POST

/api/v1/load

None

Content-Type: application/json

Parameters

Name

Type

Required

Description

Example

index

string

No

Path to the index file on UFS (must be mounted)

'{"index": "s3://bucket/key"}'

paths

string

No

List of UFS paths to load

'{"paths": ["s3://bucket/key1", "s3://bucket/key2"]}'

alias

string

No

Used in combination with paths for subsequent status queries

'{"paths": ["s3://bucket/key1"], "alias": "database1"}'

`options

object

No

Configuration options for the load job

'{"index": "s3://bucket/key", "options":{"batchSize":10}}'

Note: You must choose either index or paths. If paths is used, an alias can be specified; otherwise, a random alias will be generated and returned.

An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:

Name
Type
Required
Description

batchSize

int

No

Batch size for workers to load data from UFS

replicas

int

No

Number of file replicas to load. Defaults to the path-specific config or 1

skipIfExists

bool

No

Skip the loading process if the file already exists

loadPolicy

enum

No

Currently supports IF_CHANGED. Updates the file if it has changed

loadMetadataOnly

bool

No

Only load file metadata

Example

JSON request payload:

Success Response (JSON):

Error Codes

HTTP Code
Error Code
Description

200

OK

Operation successful

400

Bad Request

Request body missing or required fields missing

409

Conflict

An identical job is already scheduled; submission rejected

500

Internal Server Error

Unexpected error

Get Load Job Progress

Method

Path

Params

Request Body Type

GET

/api/v1/load

Yes

None

Parameters:

Name

Type

Required

Description

Example

target

string

Yes

Field information returned when submitting the job

target=job-95e8a01f-d519-4c1a-8311-26714f4db623

fileList

bool

No

Retrieve the list of loaded files

target=job-95e8a01f-d519-4c1a-8311-26714f4db623&fileList=true

fileStatus

string

No

File list type [TOTAL|FAILURE]

target=job-95e8a01f-d519-4c1a-8311-26714f4db623&fileList=true&fileStatus=TOTAL

Note: 3.7 query parameters remain compatible, eg:alias,index

Examples:

Response for job execution status:

Response for file list:

Error Codes

HTTP Code
Error Code
Description

200

OK

Operation successful

404

Not Found

Job does not exist

500

Internal Server Error

Unexpected error

Stop a load job

Method

Path

Parameters

Request Body Type

DELETE

/api/v1/load

None

Content-Type: application/json

Parameters

Name
Type
Required
Description
Example

target

string

Yes

Target ID returned at job submission

{"target":"job-95e8a01f-d519-4c1a-8311-26714f4db623"}

Note: Stop parameters remain compatible with version 3.7, eg:alias,index

Error Codes

HTTP Status
Error Code
Description

200

OK

Operation successful

400

Bad Request

No request body or missing required fields

404

Not Found

Load job does not exist

410

Gone

Load job has already completed

List all load jobs

Method

Path

Parameters

Request Body Type

GET

/api/v1/load

yes

None

Parameters:

Name

Type

Required

Description

state

enum

No

Job status [RUNNING

lastKey

string

No

List jobs starting from this key

count

int

No

Expected number of jobs to list

Note: If no parameters are provided, all jobs will be listed.

Error Codes

HTTP Status
Error Code
Description

200

OK

Operation successful

Responses Example response for listing jobs (JSON):

Free Cache

Clean up worker cache data to release storage space.

Submit or resume a free job

Method

Path

Parameters

Request Body Type

POST

/api/v1/free

None

Content-Type: application/json

Parameters

Name

Type

Required

Description

Example

index

string

No

Index file path on UFS (must be mounted)

'{"index": "s3://bucket/key"}'

paths

string

No

List of UFS paths to be freed

'{"paths": ["s3://bucket/key1", "s3://bucket/key2"]}'

alias

string

No

Used with paths for status queries

'{"paths": ["s3://bucket/key1", "s3://bucket/key2"], "alias": "database1"}'

options

object

No

Configuration options for the free job

'{"index": s3://bucket/key, "options":{"batchSize":10}}'

Note: Either index or paths must be provided. If paths is used, an alias can be specified; otherwise, an internal random alias will be generated.

At most one of the parameters is required. Paths and alias must be provided together.

An options JSON can be used to configure the options of the free job. All options are optional. Possible configs are:

Name

Type

Required

Description

batchSize

int

No

Speed at which the worker frees cache files per second

replicas

int

No

Number of file replicas to free

indexService

bool

No

Free directory cache

Example

An example JSON request payload looks like the following:

Successful Response (JSON):

Get Free Job Progress

Method

Path

Parameters

Request Body Type

GET

/api/v1/free

Yes

None

Parameters:

Name
Type
Required
Description
Example

target

string

Yes

The field returned when submitting the job

target=job-95e8a01f...

Note: 3.7 query parameters remain compatible, eg:alias,index

Example

Successful response for job status (JSON): Responses:

Error Codes

HTTP Status
Error Code
Description

200

OK

Operation successful

404

Not Found

Load job does not exist

Stop a free job

Method

Path

Parameters

Request Body Type

DELETE

/api/v1/free

None

Content-Type: application/json

Parameters

Name
Type
Required
Description
Example

target

string

Yes

The field returned when submitting the job

{"target":"job-95e8a01f-d519-4c1a-8311-26714f4db623"}

Note: Stop parameters remain compatible with version 3.7, eg:alias,index

Error Codes

HTTP Status
Error Code
Description

200

OK

Operation successful

400

Bad Request

No request body or missing required fields

404

Not Found

Job does not exist

410

Gone

Job has already completed

List all free jobs

Method

Path

Parameters

Request Body Type

GET

/api/v1/free

yes

None

Parameters:

Name

Type

Required

Description

state

enum

No

Job status [RUNNING

lastKey

string

No

List jobs starting from this key

count

int

No

Expected number of jobs to list

Note: If no parameters are provided, all jobs will be listed.

Error Codes

HTTP Status
Error Code
Description

200

OK

Operation successful

Response Example response for listing jobs (JSON):

Rebalance

List all rebalance jobs

Method: GET

Path: /api/v1/rebalance

Parameters: none

Responses:

  • 200 - OK: successful operation

    for more information on the job description, please refer to the “Get the progress of a rebalance job”

  • 500 - Internal Server Error: unexpected error

Get the progress of a rebalance job

Method: GET

Path: /api/v1/rebalance

Parameters:

At most one of the parameters is required.

Name

Description

id

job id returned by POST

target

the target workers of the rebalance job, can bea worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0)ALL (for all workers)

Responses:

  • 200 - OK: successful operation(empty value will be omitted)

  • 404 - Not Found: no job found

  • 500 - Internal Server Error: unexpected error

Submit or resume a rebalance job

Method: POST

Path: /api/v1/rebalance

Parameters: none

Request body: Content-Type: application/json

Required field

Name

Description

target

The target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)

An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:

Name

Description

loadBatchSize

the batch size in the loading phase, the same as the batchSize option in the load job

loadBandwidth

The load bandwidth of each worker per second in bytes. e.g.: 10485746000...

pruneBandwidth

The prune bandwidth of each worker per second in bytes. e.g.: 10485746000...

skipPrune

If rebalance should only load data and skip pruning the data that does not blong to the worker(s)

An example JSON request payload looks like the following:

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 409 - Conflict: resume the load job with the same path and return the previous job id

  • 500 - Internal Server Error: unexpected error

Stop a rebalance job

Method: DELETE

Path: /api/v1/rebalance

Parameters: none

Request body: Content-Type: application/json

At most one of the parameters is required.

Name

Description

id

job id returned by POST

target

the target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 410 - Gone: the job doesn’t exist or has already finished

  • 500 - Internal Server Error: unexpected error

Clear Stale Cache

This operation differs from the Free Cache operation in that:

  1. The free job requires a list of files or directories as input arguments to specify what to free from the workers. The Clear Stale Cache operation does not require such input because what counts as “stale cache” is determined automatically by scanning the worker cache storage and consulting the current consistent hash ring.

  2. The free job requires setting the correct number of replicas, otherwise it may free fewer replicas than expected, if file replication is enabled. The clear stale cache operation simply broadcasts to all the workers in the cluster, so it always triggers the clearing of stale cache on all workers, regardless of how many replicas there are.

Start clearing stale cache

Method: POST

Path: /api/v1/cache

Parameters: none

Request body: Content-Type: application/json

Responses:

  • 200 - OK: an empty errors object indicates that the operation has been successfully submitted to all workers for execution

  • 200 - OK: non-empty errors object indicates that the operation fails to be submitted to at least one of the workers for execution

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 500 - Internal Server Error: unexpected error

  • 501 - Not Implemented: the specified operation type is not implemented

Stop clearing stale cache

Method: DELETE

Path: /api/v1/cache

Parameters: none

Request body: Content-Type: application/json

Responses:

  • 200 - OK: an empty errors object indicates that all workers have been successfully notified to stop clearing stale cache

  • 200 - OK: non-empty errors object indicates that the coordinator fails to notify at least one of the workers to stop clearing stale cache

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 500 - Internal Server Error: unexpected error

  • 501 - Not Implemented: the specified operation type is not implemented

Mount Table

List mount points

Method: GET

Path: /api/v1/mount

Parameters: none

Responses:

  • 200 - OK: successful operation(options will be omitted if empty)

  • 500 - Internal Server Error: unexpected error

Get mount point information

Method: GET

Path: /api/v1/mount

Parameters:

Name

Description

path

Alluxio or UFS path of the mount point

Responses:

  • 200 - OK: successful operation(options will be omitted if empty)

  • 404 - Not Found: mount point not found

  • 500 - Internal Server Error: unexpected error

Create a mount point

Method: POST

Path: /api/v1/mount

Parameters: none

Request body: Content-Type: application/json

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 409 - Conflict: the path is already mounted, or the UFS has been mounted to another path

  • 501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)

  • 500 - Internal Server Error: unexpected error

Delete a mount point

Method: DELETE

Path: /api/v1/mount

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 410 - Gone: the path isn’t mounted to any UFS

  • 501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)

  • 500 - Internal Server Error: unexpected error

Quota

List all quota status

Method: GET

Path: /api/v1/quota

Parameters: none

Responses:

  • 200 - OK: successful operation

    Note that the returned paths are Alluxio paths.

  • 501 - Not Implemented: quota is not enabled

  • 500 - Internal Server Error: unexpected error

Get quota status

Method: GET

Path: /api/v1/quota

Parameters:

Name

Description

path

Alluxio or UFS path of the quota rule

Responses:

  • 200 - OK: successful operation

    Note that the returned paths are Alluxio paths.

  • 404 - Not Found: quota not found

  • 501 - Not Implemented: quota is not enabled

  • 500 - Internal Server Error: unexpected error

Add or update quota

Method: POST

Path: /api/v1/quota

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS

  • 501 - Not Implemented: quota is not enabled

  • 500 - Internal Server Error: unexpected error

Delete quota

Method: DELETE

Path: /api/v1/quota

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 400 - Bad Request: quota not found or other problems

  • 501 - Not Implemented: quota is not enabled

  • 500 - Internal Server Error: unexpected error

TTL

List all TTL policies

Method: GET

Path: /api/v1/ttl

Parameters: none

Responses:

  • 200 - OK: successful operation

    Note that the returned path is an Alluxio path.

  • 501 - Not Implemented: TTL policy is not enabled

  • 500 - Internal Server Error: unexpected error

Get TTL policy

Method: GET

Path: /api/v1/ttl

Parameters:

Name

Description

path

Alluxio or UFS path of the TTL policy to look up

Responses:

  • 200 - OK: successful operation

    Note that the returned path is an Alluxio path.

  • 404 - Not Found: TTL not found

  • 501 - Not Implemented: TTL policy is not enabled

  • 500 - Internal Server Error: unexpected error

Add or update TTL

Method: POST

Path: /api/v1/ttl

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS

  • 501 - Not Implemented: TTL policy is not enabled

  • 500 - Internal Server Error: unexpected error

Delete TTL

Method: DELETE

Path: /api/v1/ttl

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 400 - Bad Request: TTL not found

  • 501 - Not Implemented: TTL is not enabled

  • 500 - Internal Server Error: unexpected error

Priority Eviction

List all priority eviction policies

Method: GET

Path: /api/v1/priority

Parameters: none

Responses:

  • 200 - OK: successful operation

Note that the returned paths are Alluxio paths.

  • 501 - Not Implemented: Priority eviction is not enabled

  • 500 - Internal Server Error: unexpected error

Get priority eviction policy

Method: GET

Path: /api/v1/priority

Parameters:

Name

Description

path

Alluxio or UFS path of the priority rule to look up

Responses:

  • 200 - OK: successful operation

    Note that the returned path is an Alluxio path.

  • 404 - Not Found: priority eviction not found

  • 501 - Not Implemented: priority eviction is not enabled

  • 500 - Internal Server Error: unexpected error

Add or update priority

Method: POST

Path: /api/v1/priority

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 400 - Bad Request: priority string misformatted or other problems

  • 501 - Not Implemented: priority eviction is not enabled

  • 500 - Internal Server Error: unexpected error

Delete priority

Method: DELETE

Path: /api/v1/priority

Parameters: none

Request body: Content-Type: application/json

Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 501 - Not Implemented: priority eviction is not enabled

  • 500 - Internal Server Error: unexpected error

Node management

List all nodes

Method: GET

Path: /api/v1/nodes

Parameters: none

Responses:

  • 200 - OK: successful operation

  • 500 - Internal Server Error: unexpected error

Query a node

Method: GET

Path: /api/v1/nodes

Parameters:

Name

Description

id

worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)

Responses:

  • 200 - OK: successful operation (empty value will be omitted)

  • 404 - Not Found: worker not found

  • 500 - Internal Server Error: unexpected error

Decommission a node (remove a node from etcd)

Method: DELETE

Path: /api/v1/nodes

Parameters:

Name

Description

id

worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)

Responses:

  • 200 - OK: successful operation

  • 404 - Not Found: worker not found

  • 500 - Internal Server Error: unexpected error

Global File Index

List all nodes

Method: GET

Path: /api/v1/file_index

Parameters: none

Responses:

  • 200 - OK: successful operation

  • 500 - Internal Server Error: unexpected error

Add files into the global file index

Method: POST

Path: /api/v1/file_index

Parameters: none

Request body: Content-Type: application/json

Responses:

  • 200 - OK: successful operation (# of entries added)

  • 404 - Not Found: feature not enabled

  • 500 - Internal Server Error: unexpected error

Remove files from the global file index

Method: DELETE

Path: /api/v1/file_index

Parameters: none

Request body: Content-Type: application/json

Responses:

  • 200 - OK: successful operation (# of entries removed)

  • 404 - Not Found: feature not enabled

  • 500 - Internal Server Error: unexpected error

Cache Filter

List all cache filter rules

Method: GET

Path: /api/v1/cache-filter

Parameters: none

Responses:

  • 200 - OK: successful operation

  • 500 - Internal Server Error: unexpected error

  • 501 - Not Implemented: cache filter is not enabled

Add a cache filter rule or update the default rule

Method: POST

Path: /api/v1/cache-filter

Parameters: none

Request body: Content-Type: application/json

If the rule is maxAge, a time is required.

If you want to update the default rule, set updateDefault. No need to specify a pattern because all paths that don't match any other rules will be captured by the default rule.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 500 - Internal Server Error: unexpected error

  • 501 - Not Implemented: cache filter is not enabled

Remove a cache filter rule

Method: DELETE

Path: /api/v1/cache-filter

Parameters: none

Request body: Content-Type: application/json

No need to specify time if the rule is maxAge. Also, you can't remove a default rule. You can only change the default to another rule.

Responses:

  • 200 - OK: successful operation

  • 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  • 500 - Internal Server Error: unexpected error

  • 501 - Not Implemented: cache filter is not enabled

Last updated