REST API
The Rest APIs are designed for jobs, configurations, and cluster management.
Distributed Load
Pre-loads UFS (Underlying File System) data into worker caches to prevent performance degradation caused by cold reads.
Submit or resume a load job
Method
Path
Parameters
Request Body Type
POST
/api/v1/load
None
Content-Type: application/json
Parameters
Name
Type
Required
Description
Example
index
string
No
Path to the index file on UFS (must be mounted)
'{"index": "s3://bucket/key"}'
paths
string
No
List of UFS paths to load
'{"paths": ["s3://bucket/key1", "s3://bucket/key2"]}'
alias
string
No
Used in combination with paths for subsequent status queries
'{"paths": ["s3://bucket/key1"], "alias": "database1"}'
`options
object
No
Configuration options for the load job
'{"index": "s3://bucket/key", "options":{"batchSize":10}}'
Note: You must choose either index or paths. If paths is used, an alias can be specified; otherwise, a random alias will be generated and returned.
An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:
batchSize
int
No
Batch size for workers to load data from UFS
replicas
int
No
Number of file replicas to load. Defaults to the path-specific config or 1
skipIfExists
bool
No
Skip the loading process if the file already exists
loadPolicy
enum
No
Currently supports IF_CHANGED. Updates the file if it has changed
loadMetadataOnly
bool
No
Only load file metadata
Example
JSON request payload:
Success Response (JSON):
Error Codes
200
OK
Operation successful
400
Bad Request
Request body missing or required fields missing
409
Conflict
An identical job is already scheduled; submission rejected
500
Internal Server Error
Unexpected error
Get Load Job Progress
Method
Path
Params
Request Body Type
GET
/api/v1/load
Yes
None
Parameters:
Name
Type
Required
Description
Example
target
string
Yes
Field information returned when submitting the job
target=job-95e8a01f-d519-4c1a-8311-26714f4db623
fileList
bool
No
Retrieve the list of loaded files
target=job-95e8a01f-d519-4c1a-8311-26714f4db623&fileList=true
fileStatus
string
No
File list type [TOTAL|FAILURE]
target=job-95e8a01f-d519-4c1a-8311-26714f4db623&fileList=true&fileStatus=TOTAL
Note: 3.7 query parameters remain compatible, eg:alias,index
Examples:
Response for job execution status:
Response for file list:
Error Codes
200
OK
Operation successful
404
Not Found
Job does not exist
500
Internal Server Error
Unexpected error
Stop a load job
Method
Path
Parameters
Request Body Type
DELETE
/api/v1/load
None
Content-Type: application/json
Parameters
target
string
Yes
Target ID returned at job submission
{"target":"job-95e8a01f-d519-4c1a-8311-26714f4db623"}
Note: Stop parameters remain compatible with version 3.7, eg:alias,index
Error Codes
200
OK
Operation successful
400
Bad Request
No request body or missing required fields
404
Not Found
Load job does not exist
410
Gone
Load job has already completed
List all load jobs
Method
Path
Parameters
Request Body Type
GET
/api/v1/load
yes
None
Parameters:
Name
Type
Required
Description
state
enum
No
Job status [RUNNING
lastKey
string
No
List jobs starting from this key
count
int
No
Expected number of jobs to list
Note: If no parameters are provided, all jobs will be listed.
Error Codes
200
OK
Operation successful
Responses Example response for listing jobs (JSON):
Free Cache
Clean up worker cache data to release storage space.
Submit or resume a free job
Method
Path
Parameters
Request Body Type
POST
/api/v1/free
None
Content-Type: application/json
Parameters
Name
Type
Required
Description
Example
index
string
No
Index file path on UFS (must be mounted)
'{"index": "s3://bucket/key"}'
paths
string
No
List of UFS paths to be freed
'{"paths": ["s3://bucket/key1", "s3://bucket/key2"]}'
alias
string
No
Used with paths for status queries
'{"paths": ["s3://bucket/key1", "s3://bucket/key2"], "alias": "database1"}'
options
object
No
Configuration options for the free job
'{"index": s3://bucket/key, "options":{"batchSize":10}}'
Note: Either index or paths must be provided. If paths is used, an alias can be specified; otherwise, an internal random alias will be generated.
At most one of the parameters is required. Paths and alias must be provided together.
An options JSON can be used to configure the options of the free job. All options are optional. Possible configs are:
Name
Type
Required
Description
batchSize
int
No
Speed at which the worker frees cache files per second
replicas
int
No
Number of file replicas to free
indexService
bool
No
Free directory cache
Example
An example JSON request payload looks like the following:
Successful Response (JSON):
Get Free Job Progress
Method
Path
Parameters
Request Body Type
GET
/api/v1/free
Yes
None
Parameters:
target
string
Yes
The field returned when submitting the job
target=job-95e8a01f...
Note: 3.7 query parameters remain compatible, eg:alias,index
Example
Successful response for job status (JSON): Responses:
Error Codes
200
OK
Operation successful
404
Not Found
Load job does not exist
Stop a free job
Method
Path
Parameters
Request Body Type
DELETE
/api/v1/free
None
Content-Type: application/json
Parameters
target
string
Yes
The field returned when submitting the job
{"target":"job-95e8a01f-d519-4c1a-8311-26714f4db623"}
Note: Stop parameters remain compatible with version 3.7, eg:alias,index
Error Codes
200
OK
Operation successful
400
Bad Request
No request body or missing required fields
404
Not Found
Job does not exist
410
Gone
Job has already completed
List all free jobs
Method
Path
Parameters
Request Body Type
GET
/api/v1/free
yes
None
Parameters:
Name
Type
Required
Description
state
enum
No
Job status [RUNNING
lastKey
string
No
List jobs starting from this key
count
int
No
Expected number of jobs to list
Note: If no parameters are provided, all jobs will be listed.
Error Codes
200
OK
Operation successful
Response Example response for listing jobs (JSON):
Rebalance
List all rebalance jobs
Method: GET
Path: /api/v1/rebalance
Parameters: none
Responses:
200 - OK: successful operationfor more information on the job description, please refer to the “Get the progress of a rebalance job”
500 - Internal Server Error: unexpected error
Get the progress of a rebalance job
Method: GET
Path: /api/v1/rebalance
Parameters:
At most one of the parameters is required.
Name
Description
id
job id returned by POST
target
the target workers of the rebalance job, can bea worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0)ALL (for all workers)
Responses:
200 - OK: successful operation(empty value will be omitted)404 - Not Found: no job found500 - Internal Server Error: unexpected error
Submit or resume a rebalance job
Method: POST
Path: /api/v1/rebalance
Parameters: none
Request body: Content-Type: application/json
Required field
Name
Description
target
The target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)
An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:
Name
Description
loadBatchSize
the batch size in the loading phase, the same as the batchSize option in the load job
loadBandwidth
The load bandwidth of each worker per second in bytes. e.g.: 10485746000...
pruneBandwidth
The prune bandwidth of each worker per second in bytes. e.g.: 10485746000...
skipPrune
If rebalance should only load data and skip pruning the data that does not blong to the worker(s)
An example JSON request payload looks like the following:
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing409 - Conflict: resume the load job with the same path and return the previous job id500 - Internal Server Error: unexpected error
Stop a rebalance job
Method: DELETE
Path: /api/v1/rebalance
Parameters: none
Request body: Content-Type: application/json
At most one of the parameters is required.
Name
Description
id
job id returned by POST
target
the target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing410 - Gone: the job doesn’t exist or has already finished500 - Internal Server Error: unexpected error
Clear Stale Cache
This operation differs from the Free Cache operation in that:
The free job requires a list of files or directories as input arguments to specify what to free from the workers. The Clear Stale Cache operation does not require such input because what counts as “stale cache” is determined automatically by scanning the worker cache storage and consulting the current consistent hash ring.
The free job requires setting the correct number of replicas, otherwise it may free fewer replicas than expected, if file replication is enabled. The clear stale cache operation simply broadcasts to all the workers in the cluster, so it always triggers the clearing of stale cache on all workers, regardless of how many replicas there are.
Start clearing stale cache
Method: POST
Path: /api/v1/cache
Parameters: none
Request body: Content-Type: application/json
Responses:
200 - OK: an empty errors object indicates that the operation has been successfully submitted to all workers for execution200 - OK: non-empty errors object indicates that the operation fails to be submitted to at least one of the workers for execution400 - Bad Request: the request doesn’t have a body or the required fields are missing500 - Internal Server Error: unexpected error501 - Not Implemented: the specified operation type is not implemented
Stop clearing stale cache
Method: DELETE
Path: /api/v1/cache
Parameters: none
Request body: Content-Type: application/json
Responses:
200 - OK: an empty errors object indicates that all workers have been successfully notified to stop clearing stale cache200 - OK: non-empty errors object indicates that the coordinator fails to notify at least one of the workers to stop clearing stale cache400 - Bad Request: the request doesn’t have a body or the required fields are missing500 - Internal Server Error: unexpected error501 - Not Implemented: the specified operation type is not implemented
Mount Table
List mount points
Method: GET
Path: /api/v1/mount
Parameters: none
Responses:
200 - OK: successful operation(options will be omitted if empty)500 - Internal Server Error: unexpected error
Get mount point information
Method: GET
Path: /api/v1/mount
Parameters:
Name
Description
path
Alluxio or UFS path of the mount point
Responses:
200 - OK: successful operation(options will be omitted if empty)404 - Not Found: mount point not found500 - Internal Server Error: unexpected error
Create a mount point
Method: POST
Path: /api/v1/mount
Parameters: none
Request body: Content-Type: application/json
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing409 - Conflict: the path is already mounted, or the UFS has been mounted to another path501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)500 - Internal Server Error: unexpected error
Delete a mount point
Method: DELETE
Path: /api/v1/mount
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing410 - Gone: the path isn’t mounted to any UFS501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)500 - Internal Server Error: unexpected error
Quota
List all quota status
Method: GET
Path: /api/v1/quota
Parameters: none
Responses:
200 - OK: successful operationNote that the returned paths are Alluxio paths.
501 - Not Implemented: quota is not enabled500 - Internal Server Error: unexpected error
Get quota status
Method: GET
Path: /api/v1/quota
Parameters:
Name
Description
path
Alluxio or UFS path of the quota rule
Responses:
200 - OK: successful operationNote that the returned paths are Alluxio paths.
404 - Not Found: quota not found501 - Not Implemented: quota is not enabled500 - Internal Server Error: unexpected error
Add or update quota
Method: POST
Path: /api/v1/quota
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS501 - Not Implemented: quota is not enabled500 - Internal Server Error: unexpected error
Delete quota
Method: DELETE
Path: /api/v1/quota
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing400 - Bad Request: quota not found or other problems501 - Not Implemented: quota is not enabled500 - Internal Server Error: unexpected error
TTL
List all TTL policies
Method: GET
Path: /api/v1/ttl
Parameters: none
Responses:
200 - OK: successful operationNote that the returned path is an Alluxio path.
501 - Not Implemented: TTL policy is not enabled500 - Internal Server Error: unexpected error
Get TTL policy
Method: GET
Path: /api/v1/ttl
Parameters:
Name
Description
path
Alluxio or UFS path of the TTL policy to look up
Responses:
200 - OK: successful operationNote that the returned path is an Alluxio path.
404 - Not Found: TTL not found501 - Not Implemented: TTL policy is not enabled500 - Internal Server Error: unexpected error
Add or update TTL
Method: POST
Path: /api/v1/ttl
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS501 - Not Implemented: TTL policy is not enabled500 - Internal Server Error: unexpected error
Delete TTL
Method: DELETE
Path: /api/v1/ttl
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing400 - Bad Request: TTL not found501 - Not Implemented: TTL is not enabled500 - Internal Server Error: unexpected error
Priority Eviction
List all priority eviction policies
Method: GET
Path: /api/v1/priority
Parameters: none
Responses:
200 - OK: successful operation
Note that the returned paths are Alluxio paths.
501 - Not Implemented: Priority eviction is not enabled500 - Internal Server Error: unexpected error
Get priority eviction policy
Method: GET
Path: /api/v1/priority
Parameters:
Name
Description
path
Alluxio or UFS path of the priority rule to look up
Responses:
200 - OK: successful operationNote that the returned path is an Alluxio path.
404 - Not Found: priority eviction not found501 - Not Implemented: priority eviction is not enabled500 - Internal Server Error: unexpected error
Add or update priority
Method: POST
Path: /api/v1/priority
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing400 - Bad Request: priority string misformatted or other problems501 - Not Implemented: priority eviction is not enabled500 - Internal Server Error: unexpected error
Delete priority
Method: DELETE
Path: /api/v1/priority
Parameters: none
Request body: Content-Type: application/json
Note that the path can be either a UFS path or an Alluxio path. The path must have been mounted in Alluxio's mount table.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing501 - Not Implemented: priority eviction is not enabled500 - Internal Server Error: unexpected error
Node management
List all nodes
Method: GET
Path: /api/v1/nodes
Parameters: none
Responses:
200 - OK: successful operation500 - Internal Server Error: unexpected error
Query a node
Method: GET
Path: /api/v1/nodes
Parameters:
Name
Description
id
worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)
Responses:
200 - OK: successful operation (empty value will be omitted)404 - Not Found: worker not found500 - Internal Server Error: unexpected error
Decommission a node (remove a node from etcd)
Method: DELETE
Path: /api/v1/nodes
Parameters:
Name
Description
id
worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)
Responses:
200 - OK: successful operation404 - Not Found: worker not found500 - Internal Server Error: unexpected error
Global File Index
List all nodes
Method: GET
Path: /api/v1/file_index
Parameters: none
Responses:
200 - OK: successful operation500 - Internal Server Error: unexpected error
Add files into the global file index
Method: POST
Path: /api/v1/file_index
Parameters: none
Request body: Content-Type: application/json
Responses:
200 - OK: successful operation (# of entries added)404 - Not Found: feature not enabled500 - Internal Server Error: unexpected error
Remove files from the global file index
Method: DELETE
Path: /api/v1/file_index
Parameters: none
Request body: Content-Type: application/json
Responses:
200 - OK: successful operation (# of entries removed)404 - Not Found: feature not enabled500 - Internal Server Error: unexpected error
Cache Filter
List all cache filter rules
Method: GET
Path: /api/v1/cache-filter
Parameters: none
Responses:
200 - OK: successful operation
500 - Internal Server Error: unexpected error501 - Not Implemented: cache filter is not enabled
Add a cache filter rule or update the default rule
Method: POST
Path: /api/v1/cache-filter
Parameters: none
Request body: Content-Type: application/json
If the rule is maxAge, a time is required.
If you want to update the default rule, set updateDefault. No need to specify a pattern because all paths that don't match any other rules will be captured by the default rule.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing500 - Internal Server Error: unexpected error501 - Not Implemented: cache filter is not enabled
Remove a cache filter rule
Method: DELETE
Path: /api/v1/cache-filter
Parameters: none
Request body: Content-Type: application/json
No need to specify time if the rule is maxAge. Also, you can't remove a default rule. You can only change the default to another rule.
Responses:
200 - OK: successful operation400 - Bad Request: the request doesn’t have a body or the required fields are missing500 - Internal Server Error: unexpected error501 - Not Implemented: cache filter is not enabled
Last updated