Directory-Based Cluster Quota

Directory-Based Cluster Quota Overview

Alluxio allows admins to define a quota on a directory, which can limit the total amount of cache space used by the files in that directory in Alluxio, across all workers in the cluster.

ETCD only stores quota definitions. Current quota usages are not stored and updated in ETCD.

Workers stay updated on the latest quota definitions by querying ETCD. Based on the quota definitions, workers track the current cache usage under directory with quota definitions.

The coordinator is responsible for periodically polling the latest quota usages from the workers, and forming a cluster-wide usage view by aggregating the usage from each worker. For a certain quota definition, if the total usage in all workers exceeds the defined limit, the coordinator will detect that. Then the coordinator will send cache-control commands to all workers to handle the violated quotas. More details about cache-control commands can be found here.

Please make sure the coordinator is running when enabling directory-based cluster quota feature. If the coordinator is not running, commands that add/remove/update/list quotas will fail. The workers will keep operating, but the quota definitions will not be enforced. So some quotas may be overused without limit.

To enable the directory-based cluster quota feature, set the following properties in conf/alluxio-site.properties:

# Enable the directory-based cluster quota feature for all components
alluxio.quota.enabled=true

# Configure the coordinator address
alluxio.coordinator.address=<host>:<port>

Basic Operations

Add a quota definition

You can add a quota definition on a directory in Alluxio by specifying the directory path and the quota size. Note the path is an Alluxio path without a scheme, instead of a UFS path (like s3://bucket).

When you add a quota definition, please make sure the directory is resolvable in Alluxio. In other words, the target directory must be under an existing Alluxio mount point and it must exist in the UFS. In the example below, there are two existing mount points in Alluxio namespace, and you may only add quota definitions on directories under /s3 or /local.

# There are two existing mount points in Alluxio namespace
$ bin/alluxio mount list
Listing all mount points
s3a://alluxio-jliu/                           on  /s3/    properties={...}
file:///Documents/Alluxio/underFSStorage/     on  /local/ properties={...}

# Setting a quota on a non-existing directory will fail
$ bin/alluxio quota add --directory /another --quota-size 10GB
The specified quota path /another is not under any mount point! Existing mount points are:
[/s3/, /local/].

You can set quota definition on any directory under existing mount points. For example:

# /s3/ is an existing mount point in Alluxio namespace
# You may set a quota definition on a directory under /s3/
$ bin/alluxio quota add --directory /s3/data/ --quota-size 10GB
Successfully added quota definition for path /s3/data/ with size 10GB.

$ bin/alluxio quota list
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available

Nested quota definitions are not supported yet. You cannot add a quota definition if that will result in nested quota definitions. For example:

$ bin/alluxio quota add --directory /local/data/ --quota-size 10GB
Failed to add quota definition: UNKNOWN: New quota /local/data already has a parent quota /local. Nested quota definitions are not supported in this version.

Remove a quota definition

You can remove a quota definition on a directory in Alluxio by specifying the directory path.

$ bin/alluxio quota remove --directory /s3/data
Successfully removed quota definition for path /s3/data.

Update a quota definition

Updating a quota definition takes the same arguments as adding a quota definition.

$ bin/alluxio quota update --directory /local/data/ --quota-size 100GB

However, note that if you are updating the quota size to a smaller value than the current usage, the update will fail. In this case, we recommend freeing up some space before reducing the quota size.

$ bin/alluxio quota update --directory /local --quota-size 1MB
Loading the latest quota definitions from ETCD
Latest quota definitions loaded: {
/local=alluxioPath: "/local/"
quota: 11534336
ufsPath: "file:///Documents/Alluxio/underFSStorage/"
}
Reducing quota size for path /local from 11534336 to 1048576.
Failed to update quota: RESOURCE_EXHAUSTED: The target quota size 1048576 is smaller than the current usage 10486645. Please free up some space before reducing the quota size. Or use the --force option.

In that case, you may use the --force option to force the update. However, that is not the recommended approach. If you force-update the quota capacity to be less than the current usage, the coordinator will realize the quota is violated. It will then trigger eviction on all workers to evict some cache under that quota. It may also, depending on the configuration, ask the workers to either stop caching or reject I/O requests if they try to create new cache under that directory.

$ bin/alluxio quota update --directory /local --quota-size 1MB --force

List current quota definitions and usages

You can list all existing quota definitions and their usages in Alluxio by running the following command. Note that if you have just added a new quota definition, workers may need some time to scan their existing cache and calculate the current usage under this directory. In the meantime, the usage will be marked as Calculating.

$ bin/alluxio quota list
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available

Alluxio also provides a command to continuously poll the latest quota usages from the coordinator:

# Keeps running until interrupted by user by pressing ctrl+C
$ bin/alluxio quota list --interval 5s
Polling quota usage summary from the master every 5s
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: Calculating, State: Available
Alluxio path: /local, Quota capacity bytes: 11534336, Used bytes: 10486645, State: Available
Alluxio path: /s3/data, Quota capacity bytes: 10737418240, Used bytes: 10486645, State: Available

Advanced Configuration

When Quota Limit Is Exceeded

When the current quota usage exceeds the defined limit, the coordinator will command all workers in the cluster to evict some cache under that quota, so the aggregated usage will fall below the limit again. For example, if there is one existing quota definition of 10GB capacity on Alluxio path /s3/ with 2 workers in the cluster.

Worker A currently holds 12GB cache under /s3/.
Worker B currently holds 4GB cache under /s3/.

The coordinator will aggregate the total usage from both workers, and observe that the current total is 16GB, which exceeds the limit of 10GB. The coordinator will send a command to both worker A and B, asking them to each evict (16 - 10) / 16 = 0.375 of their current cache under /s3/. If the eviction is successful, after a short time, the cache usage of the two workers will become:

Worker A now holds 7.5GB cache under /s3/.
Worker B now holds 2.5GB cache under /s3/.

As can be observed, when the quota limit is exceeded, the coordinator will ask each worker to evict the same percentage of cache. This is the only eviction strategy supported.

On top of eviction, alluxio.quota.limit.exceeded.action controls the behavior when a quota is violated. There are 3 supported modes:

NO_CACHE (default): When a quota is violated, all workers will avoid adding new cache under that path. If a read request is a cache-miss, the worker will serve it by reading the UFS but not caching. This mode stops new cache from being added to the cluster, while allowing requests to complete without errors.
REJECT: When a quota is violated, all workers will reject new cache requests under that path. If a request wants to create new cache under that path, the worker returns an exception. The behavior of Alluxio cluster cache becomes more similar to a disk, where you cannot write into a full disk.
NOOP: When a quota is violated, still allow new cache to be created under that path. Under this mode, new cache will keep being added to the workers, whereas the coordinator keeps commanding workers to evict cache to restore the quota limit. Note that if the incoming rate of cache is faster than eviction, the usage may never be restored to below the limit set by the quota definition.

Quota Coordinator Heartbeat Interval

alluxio.quota.worker.heartbeat.interval.ms controls the frequency of coordinator aggregating the current quota usage from workers in the cluster. Because the coordinator polls the quota usages from workers periodically, the quota usage it observes (also reflected in the bin/alluxio quota list command) will have a delay.

# Default value is 1s
alluxio.quota.worker.heartbeat.interval.ms=1s

Quota in Load Jobs

The Load command will load a directory or a large list of files specified in an index file into the cache. It is possible for a load operation to exceed a quota and cause undesired cache eviction. To avoid this scenario, a check can be enabled to reject the job if the load operation would exceed a quota.

This check will calculate and compare 3 pieces of information:

The total size of the files in the UFS
For those files, how much of them are already cached in Alluxio workers
The current availability of the quota definition

For example, if the index file specifies 100 files of total 100GB size, with 50GB of them are already cached in the workers and the quota currently only has 10GB availability left, Alluxio should reject this load job. The admin should manually free up at least 40GB from that quota before trying to submit the load job again.

Constraints and Configurations

Enable Quota Check Step on Load

By default, the quota check step in a Load job is disabled and none of the following constraints will be enforced. To enable the quota check step, set the following property:

# false by default
alluxio.dora.load.job.must.check.quota=true

Adding the --skip-quota-check will override the quota check enabled by the configuration property.

# Skip the quota check and start loading directly 
$ ./bin/alluxio job load --path s3://alluxio-test/data/ --submit --skip-quota-check

The quota check must be enabled for any of the following constraints to be enforced. If it is disabled by configuration or command line flag, no checks will occur.

Prevent Loading Files from Multiple Quota Definitions

The quota check will reject a load job if it involves files from multiple quota definitions. This applies to both loading a directory or loading a list of files specified in an index file. This constraint will always be enforced if quota check is enabled.

Enforce Quota Definition for All Load Jobs

The quota check can be configured to reject a load job if it tries to load a path that is not under a quota definition. This constraint can be used to ensure all load jobs are under the quota control. It can optionally be turned off from the set of quota checks by setting:

# true by default
alluxio.dora.load.job.without.quota.allowed=false

Prevent Concurrent Load Jobs Under One Quota Definition

Since the quota is only checked at the start of a load job, it is possible to exceed the quota if running concurrent load jobs under the same quota. To avoid this situation, the quota check can be configured to only allow one load job at a time to run within a path set with a quota. To opt into enforcing this restriction, set the property:

# false by default
alluxio.dora.load.job.quota.mutual.exclusive=true

Last updated 5 days ago