REST API

Coordinator

The APIs on the coordinator are designed for jobs, configurations, and cluster management.

Distributed Load

List all load jobs

Method: GET

Path: /api/v1/load

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {...}
    ]
  }

  for more information on the job description, please refer to the “Get the progress of a load job”

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get the progress of a load job

Method: GET

Path: /api/v1/load

Parameters:

Name	Description
id	job id returned by POST

Responses:

• 200 - OK: successful operation(empty value will be omitted)

  {
    "id": "54b88939-de49-46ef-acab-fe489f46d1a0",
    "startTime": "2024-11-21T20:58:26.957+08:00",
    "state": "SUCCEEDED",
    "runningState": "",
    "errorMessage": {},
    "failedFileListPath": "",
    "paths": [
      "s3://my-bucket/dir/to/path"
    ],
    "index": "",
    "skipIfExists": false,
    "loadMetadataOnly": false,
    "verify": false,
    "bandwidth": 0,
    "timeElapsedMilliseconds": 10949,
    "loadedBytes": 10485760,
    "skippedBytes": 0,
    "totalBytes": 10485760,
    "loadedNonEmptyFiles": 1,
    "failedFiles": 0,
    "scannedFiles": 1,
    "quota": {
      "totalBytes": 0,
      "remaining": 0
    }
    "quotaCheckError": ""
  }

• 404 - Not Found: no job found

  {
    "status": "No job found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Submit or resume a load job

Method: POST

Path: /api/v1/load

Parameters: none

Request body: Content-Type: application/json

At most one of the parameters is required.

Name	Description
paths	a list of ufs paths to load
index	the path of an index file on ufs (must be mounted)

An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:

Name	Description
batchSize	the batch size for workers to load the data from ufs
fileFilterRegex	a regex to filter files to load
replicas	# of replicas to load, the default value is 1. If multiple replicas is enabled, can specify this argument to match the # of replicas
skipIfExists	if the file already exists, skip the loading process
sync	Whether the job needs to run synchronously and obtain the execution results

An example JSON request payload looks like the following:

{
  "paths": [
    "s3://my-bucket/dir/to/path"
  ],
  "options": {
    "batchSize": 0,
    "fileFilterRegx": "",
    "replicas": 0,
    "sync": true,
    "skipIfExists": false
  }
}

Responses:

• 200 - OK: successful operation

  {
    "id": "54b88939-de49-46ef-acab-fe489f46d1a0"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 409 - Conflict: resume the load job with the same path and return the previous job id

  {
    "id": "54b88939-de49-46ef-acab-fe489f46d1a0"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Stop a load job

Method: DELETE

Path: /api/v1/load

Parameters: none

Request body: Content-Type: application/json

{
  "id": "54b88939-de49-46ef-acab-fe489f46d1a0"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 410 - Gone: the job doesn’t exist or has already finished

  {
    "status": "The job doesn't exist or has already finished"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Free Cache

List all free jobs

Method: GET

Path: /api/v1/free

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {...}
    ]
  }

  for more information on the job description, please refer to the “Get the progress of a free job”

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get the progress of a free job

Method: GET

Path: /api/v1/free

Parameters:

Name	Description
id	job id returned by POST

Responses:

• 200 - OK: successful operation(empty value will be omitted)

  {
    "id": "464050c3-9900-4b2f-8716-9220cd5eb495",
    "startTime": "2024-11-21T21:21:55.764+08:00",
    "state": "SUCCEEDED",
    "errorMessage": {},
    "paths": [
      "s3://my-bucket/dir/to/path"
    ],
    "timeElapsedMilliseconds": 4197,
    "freedBytes": 10485760,
    "failedBytes": 0,
    "totalBytes": 10485760,
    "freedFiles": 1,
    "failedFiles": 0,
    "totalFiles": 10485760
  }

• 404 - Not Found: no job found

  {
    "status": "No job found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Submit or resume a free job

Method: POST

Path: /api/v1/free

Parameters: none

Request body: Content-Type: application/json

At most one of the parameters is required.

Name	Description
paths	a list of ufs paths to load
index	the path of an index file on ufs (must be mounted)

An example JSON request payload looks like the following:

{
  "paths": [
    "s3://my-bucket/dir/to/path"
  ],
  "index": "",
  "options": {
    "batchSize": 0,
    "sync": true,
    "recursive": true
  }
}

Responses:

• 200 - OK: successful operation

  {
    "id": "464050c3-9900-4b2f-8716-9220cd5eb495"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 409 - Conflict: resume the free job with the same path and return the previous job id

  {
    "id": "464050c3-9900-4b2f-8716-9220cd5eb495"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Stop a free job

Method: DELETE

Path: /api/v1/free

Parameters: none

Request body: Content-Type: application/json

{
  "id": "464050c3-9900-4b2f-8716-9220cd5eb495"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 410 - Gone: the job doesn’t exist or has already finished

  {
    "status": "The job doesn't exist or has already finished"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Rebalance

List all rebalance jobs

Method: GET

Path: /api/v1/rebalance

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {...}
    ]
  }

  for more information on the job description, please refer to the “Get the progress of a rebalance job”

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get the progress of a rebalance job

Method: GET

Path: /api/v1/rebalance

Parameters:

At most one of the parameters is required.

Name	Description
id	job id returned by POST
target	the target workers of the rebalance job, can bea worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0)ALL (for all workers)

Responses:

• 200 - OK: successful operation(empty value will be omitted)

  {
     "succeededWorkers":2,
     "bytesPreserved":4,
     "filesLoaded":0,
     "filesPreserved":1,
     "filesFailed":0,
     "totalWorkers":2,
     "bytesPruned":226,
     "filesPruned":5,
     "bytesLoaded":0,
     "timeElapsedMilliseconds":4536,
     "runningWorkers":0,
     "startTime":"2024-12-19T23:15:07.043+08:00",
     "id":"8de63d3a-63c8-4f08-bbc1-c87e3c0fb6d9",
     "state":"SUCCEEDED",
     "failedWorkers":0
  }

• 404 - Not Found: no job found

  {
    "status": "No job found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Submit or resume a rebalance job

Method: POST

Path: /api/v1/rebalance

Parameters: none

Request body: Content-Type: application/json

Required field

Name	Description
target	The target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)

An options JSON can be used to configure the options of the load job. All options are optional. Possible configs are:

Name	Description
loadBatchSize	the batch size in the loading phase, the same as the batchSize option in the load job
loadBandwidth	The load bandwidth of each worker per second in bytes. e.g.: 10485746000...
pruneBandwidth	The prune bandwidth of each worker per second in bytes. e.g.: 10485746000...
skipPrune	If rebalance should only load data and skip pruning the data that does not blong to the worker(s)

An example JSON request payload looks like the following:

{
  "target": "ALL",
  "options": {
    "loadBatchSize": 1000,
    "loadBandwidth": 10485746000,
    "pruneBandwidth": 10485746000,
    "skipPrune": false,
  }
}

Responses:

• 200 - OK: successful operation

  {
    "id": "54b88939-de49-46ef-acab-fe489f46d1a0"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 409 - Conflict: resume the load job with the same path and return the previous job id

  {
    "id": "54b88939-de49-46ef-acab-fe489f46d1a0"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Stop a rebalance job

Method: DELETE

Path: /api/v1/rebalance

Parameters: none

Request body: Content-Type: application/json

At most one of the parameters is required.

Name	Description
id	job id returned by POST
target	the target workers of the rebalance job, can be either a worker id (e.g. worker-54b88939-de49-46ef-acab-fe489f46d1a0) or ALL (for all workers)

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 410 - Gone: the job doesn’t exist or has already finished

  {
    "status": "The job doesn't exist or has already finished"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Clear Stale Cache

This operation differs from the Free Cache operation in that:

1. The free job requires a list of files or directories as input arguments to specify what to free from the workers. The Clear Stale Cache operation does not require such input because what counts as “stale cache” is determined automatically by scanning the worker cache storage and consulting the current consistent hash ring.

2. The free job requires setting the correct number of replicas, otherwise it may free fewer replicas than expected, if file replication is enabled. The clear stale cache operation simply broadcasts to all the workers in the cluster, so it always triggers the clearing of stale cache on all workers, regardless of how many replicas there are.

Start clearing stale cache

Method: POST

Path: /api/v1/cache

Parameters: none

Request body: Content-Type: application/json

{
  "op": "clear-stale"
}

Responses:

• 200 - OK: an empty errors object indicates that the operation has been successfully submitted to all workers for execution

  {
    "errors": {}
  }

• 200 - OK: non-empty errors object indicates that the operation fails to be submitted to at least one of the workers for execution

  {
    "errors": {
      "worker-host": "error message"
    }
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

• 501 - Not Implemented: the specified operation type is not implemented

  {
    "status": "OP not implemented"
  }

Stop clearing stale cache

Method: DELETE

Path: /api/v1/cache

Parameters: none

Request body: Content-Type: application/json

{
  "op": "clear-stale"
}

Responses:

• 200 - OK: an empty errors object indicates that all workers have been successfully notified to stop clearing stale cache

  {
    "errors": {}
  }

• 200 - OK: non-empty errors object indicates that the coordinator fails to notify at least one of the workers to stop clearing stale cache

  {
    "errors": {
      "worker-host": "error message"
    }
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

• 501 - Not Implemented: the specified operation type is not implemented

  {
    "status": "OP not implemented"
  }

Mount Table

List mount points

Method: GET

Path: /api/v1/mount

Parameters: none

Responses:

• 200 - OK: successful operation(options will be omitted if empty)

  {
    "results": [
      {
        "path": "/data/",
        "ufs": "s3://my-bucket/",
        "options": {
          "s3a.secretKey": "string",
          "s3a.accessKeyId": "string"
        }
      }
    ]
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get mount point information

Method: GET

Path: /api/v1/mount

Parameters:

Name	Description
path	alluxio path of the mount point

Responses:

• 200 - OK: successful operation(options will be omitted if empty)

  {
    "path": "/data/",
    "ufs": "s3://my-bucket/",
    "options": {
      "s3a.secretKey": "string",
      "s3a.accessKeyId": "string"
    }
  }

• 404 - Not Found: mount point not found

  {
    "status": "No mount point found for path: /data/"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Create a mount point

Method: POST

Path: /api/v1/mount

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data",
  "ufs": "s3://my-bucket",
  "options": {
    "s3a.secretKey": "string",
    "s3a.accessKeyId": "string"
  }
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 409 - Conflict: the path is already mounted, or the UFS has been mounted to another path

  {
    "status": "string"
  }

• 501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)

  {
    "status": "string"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Delete a mount point

Method: DELETE

Path: /api/v1/mount

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 410 - Gone: the path isn’t mounted to any UFS

  {
    "status": "string"
  }

• 501 - Not Implemented: the mount table doesn’t support managing via an API (e.g.: static file mount table)

  {
    "status": "string"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Quota

List all quota status

Method: GET

Path: /api/v1/quota

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {
        "path": "/data/dir/to/path",
        "state": "Available",
        "quotaBytes": 10737418240,
        "usedBytes": 9663676416,
        "reservedBytes": 1073741824
      }
    ]
  }

• 501 - Not Implemented: quota is not enabled

  {
    "status": "Quota not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get quota status

Method: GET

Path: /api/v1/quota

Parameters:

Name	Description
path	alluxio path to the data

Responses:

• 200 - OK: successful operation

  {
    "path": "/data/dir/to/path",
    "state": "Available",
    "quotaBytes": 10737418240,
    "usedBytes": 9663676416,
    "reservedBytes": 1073741824
  }

• 404 - Not Found: quota not found

  {
    "status": "No quota item found for path: /data/dir/to/path"
  }

• 501 - Not Implemented: quota is not enabled

  {
    "status": "Quota not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Add or update quota

Method: POST

Path: /api/v1/quota

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path",
  "quotaBytes": 10737418240
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS

  {
    "status": "No mounted UFS found"
  }

• 501 - Not Implemented: quota is not enabled

  {
    "status": "Quota not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Delete quota

Method: DELETE

Path: /api/v1/quota

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 400 - Bad Request: quota not found or other problems

  {
    "status": "string"
  }

• 501 - Not Implemented: quota is not enabled

  {
    "status": "Quota not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

TTL

List all TTL policies

Method: GET

Path: /api/v1/ttl

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {
        "path": "/data/dir/to/path",
        "ttlSeconds": 600
      }
    ]
  }

• 501 - Not Implemented: TTL policy is not enabled

  {
    "status": "TTL not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get TTL policy

Method: GET

Path: /api/v1/ttl

Parameters:

Name	Description
path	alluxio path to the data

Responses:

• 200 - OK: successful operation

  {
    "path": "/data/dir/to/path",
    "ttlSeconds": 600
  }

• 404 - Not Found: TTL not found

  {
    "status": "No TTL policy found for path: /data/dir/to/path"
  }

• 501 - Not Implemented: TTL policy is not enabled

  {
    "status": "TTL not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Add or update TTL

Method: POST

Path: /api/v1/ttl

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path",
  "ttlSeconds": 600
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 400 - Bad Request: the path doesn’t refer to a mounted UFS but it must be bound with an existing UFS

  {
    "status": "No mounted UFS found"
  }

• 501 - Not Implemented: TTL policy is not enabled

  {
    "status": "TTL not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Delete TTL

Method: DELETE

Path: /api/v1/ttl

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 400 - Bad Request: TTL not found

  {
    "status": "string"
  }

• 501 - Not Implemented: TTL is not enabled

  {
    "status": "TTL not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Priority Eviction

List all priority eviction policies

Method: GET

Path: /api/v1/priority

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      {
        "path": "/data/dir/to/path",
        "priority": "high"
      }
    ]
  }

• 501 - Not Implemented: Priority eviction is not enabled

  {
    "status": "Priority eviction not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Get priority eviction policy

Method: GET

Path: /api/v1/priority

Parameters:

Name	Description
path	alluxio path to the data

Responses:

• 200 - OK: successful operation

  {
    "path": "/data/dir/to/path",
    "priority": "high"
  }

• 404 - Not Found: priority eviction not found

  {
    "status": "No priority rule found for path: /data/dir/to/path"
  }

• 501 - Not Implemented: priority eviction is not enabled

  {
    "status": "Priority eviction not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Add or update priority

Method: POST

Path: /api/v1/priority

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path",
  "priority": "high"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 400 - Bad Request: priority string misformatted or other problems

  {
    "status": "string"
  }

• 501 - Not Implemented: priority eviction is not enabled

  {
    "status": "Priority eviction not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Delete priority

Method: DELETE

Path: /api/v1/priority

Parameters: none

Request body: Content-Type: application/json

{
  "path": "/data/dir/to/path"
}

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 400 - Bad Request: the request doesn’t have a body or the required fields are missing

  {
    "status": "Required parameters not provided"
  }

• 501 - Not Implemented: priority eviction is not enabled

  {
    "status": "Priority eviction not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Node management

List all nodes

Method: GET

Path: /api/v1/nodes

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "result": [
      {
        "identity": "worker-587899dd-5da5-45d4-af40-ce481acc4087",
        "address": "192.168.1.19:29989",
        "status": "ONLINE"
      },
      {
        "identity": "worker-844db66c-d7fd-46cc-8de0-2f3989360828",
        "address": "192.168.1.19:29999",
        "status": "OFFLINE"
      }
    ]
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Query a node

Method: GET

Path: /api/v1/nodes

Parameters:

Name	Description
id	worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)

Responses:

• 200 - OK: successful operation (empty value will be omitted)

  {
    "identity": "worker-587899dd-5da5-45d4-af40-ce481acc4087",
    "address": "192.168.1.19:29989",
    "status": "ONLINE"
  }

• 404 - Not Found: worker not found

  {
    "status": "worker not found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Decommission a node (remove a node from etcd)

Method: DELETE

Path: /api/v1/nodes

Parameters:

Name	Description
id	worker id (e.g. worker-587899dd-5da5-45d4-af40-ce481acc4087)

Responses:

• 200 - OK: successful operation

  {
    "status": "ok"
  }

• 404 - Not Found: worker not found

  {
    "status": "worker not found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Global File Index

List all nodes

Method: GET

Path: /api/v1/file_index

Parameters: none

Responses:

• 200 - OK: successful operation

  {
    "results": [
      "file:///home/yimin/alluxio/ufs/10MB",
      "file:///home/yimin/alluxio/ufs/to_load/f1"
    ]
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Add files into the global file index

Method: POST

Path: /api/v1/file_index

Parameters: none

Request body: Content-Type: application/json

{
  "files": [
    "s3://foo/bar/f1",
    "s3://foo/bar/f2"
  ]
}

Responses:

• 200 - OK: successful operation (# of entries added)

  {
    "entries": 10
  }

• 404 - Not Found: feature not enabled

  {
    "status": "Global file index is not enabled"
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }

Remove files from the global file index

Method: DELETE

Path: /api/v1/file_index

Parameters: none

Request body: Content-Type: application/json

{
  "files": [
    "s3://foo/bar/f1",
    "s3://foo/bar/f2"
  ]
}

Responses:

• 200 - OK: successful operation (# of entries removed)

  {
    "entries": 10
  }

• 404 - Not Found: feature not enabled

  {
    "status": "worker not found..."
  }

• 500 - Internal Server Error: unexpected error

  {
    "status": "string"
  }