Directory-Based Cluster Quota

Directory-Based Cluster Quota Overview

Alluxio allows admins to define a quota on a directory, which can limit the total amount of cache space used by the files in that directory in Alluxio, across all workers in the cluster.

ETCD only stores quota definitions. Current quota usages are not stored and updated in ETCD.

Workers stay updated on the latest quota definitions by querying ETCD. Based on the quota definitions, workers track the current cache usage under directory with quota definitions.

The coordinator is responsible for periodically polling the latest quota usages from the workers, and forming a cluster-wide usage view by aggregating the usage from each worker. For a certain quota definition, if the total usage in all workers exceeds the defined limit, the coordinator will detect that. Then the coordinator will send cache-control commands to all workers to handle the violated quotas. More details about cache-control commands can be found here.

Please make sure the coordinator is running when enabling directory-based cluster quota feature. If the coordinator is not running, commands that add/remove/update/list quotas will fail. The workers will keep operating, but the quota definitions will not be enforced. So some quotas may be overused without limit.

To enable the directory-based cluster quota feature, set the following properties in conf/alluxio-site.properties:

# Enable the directory-based cluster quota feature for all components
alluxio.quota.enabled=true

# Configure the coordinator address
alluxio.coordinator.address=<host>:<port>

Basic Operations

Add a quota definition

You can add a quota definition on a directory in Alluxio by specifying the directory path and the quota size. Note the path is an Alluxio path without a scheme, instead of a UFS path (like s3://bucket).

When you add a quota definition, please make sure the directory is resolvable in Alluxio. In other words, the target directory must be under an existing Alluxio mount point and it must exist in the UFS. In the example below, there are two existing mount points in Alluxio namespace, and you may only add quota definitions on directories under /s3 or /local.

# There are two existing mount points in Alluxio namespace
$ bin/alluxio mount list
Listing all mount points
s3a://alluxio-jliu/                           on  /s3/    properties={...}
file:///Documents/Alluxio/underFSStorage/     on  /local/ properties={...}

# Setting a quota on a non-existing directory will fail
$ bin/alluxio quota add --directory /another --quota-size 10GB
The specified quota path /another is not under any mount point! Existing mount points are:
[/s3/, /local/].

You can set quota definition on any directory under existing mount points. For example:

# /s3/ is an existing mount point in Alluxio namespace
# You may set a quota definition on a directory under /s3/
$ bin/alluxio quota add --directory /s3/data/ --quota-size 10GB
Successfully added quota definition for path /s3/data/ with size 10GB.

$ bin/alluxio quota list
Alluxio path                    	        Capacity	            Used	State
/local                          	       1024.00MB	        900.10MB	Available
/s3/data                    	             10.00GB	     Calculating	Available

Alluxio also supports nested quota definitions. For example, you can add children quota definitions under an existing parent quota definition, to further split a quota into smaller ones.

$ bin/alluxio quota add --directory /s3/data/ --quota-size 10GB
Successfully added quota definition for path /s3/data/ with size 10GB.

$ bin/alluxio quota add --directory /s3/data/team1 --quota-size 5GB
Successfully added quota definition for path /s3/data/team1 with size 5GB.

$ bin/alluxio quota add --directory /s3/data/team2 --quota-size 4GB
Successfully added quota definition for path /s3/data/team2 with size 4GB.

$ bin/alluxio quota add --directory /s3/data/team1/user1 --quota-size 2GB
Successfully added quota definition for path /s3/data/team1/user1 with size 2GB.

The sum of children quota definitions must not exceed the parent quota definition. We also highly recommend leaving enough buffer in the parent quota definition, which is not allocated to any child quota definitions.

# Suppose you have an existing parent and a child quota definition
$ bin/alluxio quota add --directory /s3/data/ --quota-size 10GB
$ bin/alluxio quota add --directory /s3/data/user1 --quota-size 5GB

# Adding the new child quota definition below will fail, as the sum of children would exceed the parent
$ bin/alluxio quota add --directory /s3/data/user2 --quota-size 6GB
Failed to add quota definition: 
INVALID_ARGUMENT: Invalid quota definition:
Parent quota: (/s3/data : 10.00GB)
Current children quotas: [(/s3/data/user1 : 5.00GB)]
Adding this new quota will exceed the parent quota.

You can add a parent quota definition on top of existing children quota definitions.

$ bin/alluxio quota add --directory /s3/data/team1 --quota-size 5GB
Successfully added quota definition for path /s3/data/team1 with size 5GB.

$ bin/alluxio quota add --directory /s3/data/team2 --quota-size 4GB
Successfully added quota definition for path /s3/data/team2 with size 4GB.

# The parent quota must be larger than sum of all children quotas
$ bin/alluxio quota add --directory /s3/data/ --quota-size 10GB
Successfully added quota definition for path /s3/data/ with size 10GB.

You can also add a child quota definition at a disjoint level. In other words, not every directory level in a quota definition needs to have a quota definition.

$ bin/alluxio quota add --directory /s3/data/shared/past-records --quota-size 512MB
Successfully added quota definition for path /s3/data/shared/past-records with size 512MB.

# The sum of all children quota definitions still does not exceed the parent quota definition
$ bin/alluxio quota list
Alluxio path                    	        Capacity	            Used	State
/s3/data/                          	         10.00GB	        464.00MB	Available
/s3/data/shared/past-records           	    512.00MB	              0B	Available
/s3/data/team1                    	          3.00GB	              0B	Available
/s3/data/team2                    	          3.00GB	              0B	Available

There is no limit in how many levels of directory a quota definition can have. There is also no limit in how many children a parent quota definition can have. However, we do not recommend defining more than 100 quota rules or more than 3 levels of nested quota rules. That will both introduce performance overhead to the cluster and operational overhead to the cluster administrator.

Remove a quota definition

You can remove a quota definition on a directory in Alluxio by specifying the directory path.

$ bin/alluxio quota remove --directory /s3/data
Successfully removed quota definition for path /s3/data.

You can remove a parent quota definition without removing its children first.

Update a quota definition

Updating a quota definition takes the same arguments as adding a quota definition.

$ bin/alluxio quota update --directory /local/data/ --quota-size 100GB

However, note that if you are updating the quota size to a smaller value than the current usage, the update will fail. In this case, we recommend freeing up some space before reducing the quota size.

$ bin/alluxio quota update --directory /local --quota-size 1MB
Loading the latest quota definitions from ETCD
Latest quota definitions loaded: {
/local=alluxioPath: "/local/"
quota: 11534336
ufsPath: "file:///Documents/Alluxio/underFSStorage/"
}
Reducing quota size for path /local from 11534336 to 1048576.
Failed to update quota: RESOURCE_EXHAUSTED: The target quota size 1048576 is smaller than the current usage 10486645. Please free up some space before reducing the quota size. Or use the --force option.

In that case, you may use the --force option to force the update. However, that is not the recommended approach. If you force-update the quota capacity to be less than the current usage, the coordinator will realize the quota is violated. It will then trigger eviction on all workers to evict some cache under that quota. It may also, depending on the configuration, ask the workers to either stop caching or reject I/O requests if they try to create new cache under that directory. See more details about the configured action in When Quota Limit Is Exceeded

$ bin/alluxio quota update --directory /local --quota-size 1MB --force

When you update a quota definition, the sum of all children quota definitions may never exceed the parent quota definition. If the updated quota definition violates this rule, the update will fail.

List current quota definitions and usages

You can list all existing quota definitions and their usages in Alluxio by running the following command. Note that if you have just added a new quota definition, workers may need some time to scan their existing cache and calculate the current usage under this directory. In the meantime, the usage will be marked as Calculating. A child quota usage will be reported under its parent quota.

$ bin/alluxio quota list
Alluxio path                    	        Capacity	            Used	State
/local                          	       1024.00MB	              0B	Available
/s3/data                    	             10.00GB	              0B	Available
/s3/data/user1                    	          5.00GB	              0B	Available
/s3/data/user2                    	          4.00GB	              0B	Available

Alluxio also provides a command to continuously poll the latest quota usages from the coordinator:

# Keeps running until interrupted by user by pressing ctrl+C
$ bin/alluxio quota list --interval 5s
Polling quota usage summary from the master every 5s
Alluxio path                    	        Capacity	            Used	State
/local                          	       1024.00MB	              0B	Available
/s3/data                    	             10.00GB	              0B	Available
/s3/data/user1                    	          5.00GB	              0B	Available
/s3/data/user2                    	          4.00GB	              0B	Available
Alluxio path                    	        Capacity	            Used	State
/local                          	       1024.00MB	         10.00MB	Available
/s3/data                    	             10.00GB	        100.00MB	Available
/s3/data/user1                    	          5.00GB	        100.00MB	Available
/s3/data/user2                    	          4.00GB	              0B	Available
...

Advanced Configuration and Operations

When Quota Limit Is Exceeded

When the current quota usage exceeds the defined limit, the coordinator will command all workers in the cluster to evict some cache under that quota, so the aggregated usage will fall below the limit again. For example, if there is one existing quota definition of 10GB capacity on Alluxio path /s3/ with 2 workers in the cluster.

Worker A currently holds 12GB cache under /s3/.
Worker B currently holds 4GB cache under /s3/.

The coordinator will aggregate the total usage from both workers, and observe that the current total is 16GB, which exceeds the limit of 10GB. The coordinator will send a command to both worker A and B, asking them to each evict (16 - 10) / 16 = 0.375 of their current cache under /s3/. If the eviction is successful, after a short time, the cache usage of the two workers will become:

Worker A now holds 7.5GB cache under /s3/.
Worker B now holds 2.5GB cache under /s3/.

As can be observed, when the quota limit is exceeded, the coordinator will ask each worker to evict the same percentage of cache. This is the only eviction strategy supported.

On top of eviction, alluxio.quota.limit.exceeded.action controls the behavior when a quota is violated. There are 3 supported modes:

NO_CACHE (default): When a quota is violated, all workers will avoid adding new cache under that path. If a read request is a cache-miss, the worker will serve it by reading the UFS but not caching. This mode stops new cache from being added to the cluster, while allowing requests to complete without errors.
REJECT: When a quota is violated, all workers will reject new cache requests under that path. If a request wants to create new cache under that path, the worker returns an exception. The behavior of Alluxio cluster cache becomes more similar to a disk, where you cannot write into a full disk.
NOOP: When a quota is violated, still allow new cache to be created under that path. Under this mode, new cache will keep being added to the workers, whereas the coordinator keeps commanding workers to evict cache to restore the quota limit. Note that if the incoming rate of cache is faster than eviction, the usage may never be restored to below the limit set by the quota definition.

Eviction for Nested Quota Definitions

When the cache usage under a quota definition is above the limit, cache eviction will be triggered to restore the quota limit. The behavior of eviction is more complicated when there are nested quota definitions.

Within one group of parent and child quota definitions, the quota usages can be over the limit in cases below:

Case 1: Only child quotas are overused One or more child quotas are overused, but the usage of parent quota is still below the limit. In this case, eviction will be triggered only on the overused child quotas. As a consequence, some cache in the corresponding child quota directories will be evicted.

Case 2: Only the parent quota is overused No child quotas are overused, but the usage of parent quota is above the limit. In this case, eviction will be triggered on the parent quota directory. The eviction will consider all cache under the parent directory according to the eviction policy, including those in the child quota directories. As a consequence, quota usage in some child directories may decrease, although their quota capacity are not exceeded.

Case 3: The parent quota is overused because of overuse in child quotas One or more child quotas are overused. That causes the parent quota usage to be over the capacity. In this case, eviction will be triggered only on the overused child quotas. When usages in those child quotas are restored to below the capacity, the parent quota usage will also go back to below the limit.

Case 4: Both the parent and child quotas are overused One or more child quotas are overused, but the overuse in the parent quota is more than that. In this case, the evaluation of which files to evict starts from the innermost level of quota definitions. Eviction for parent quota definitions are not considered until the quotas for their children are met. For example, the parent quota usage is 100GB over the capacity, and one child quota is 50GB over the capacity. Then 50GB will be first evicted from this child quota, and another 50GB will be evicted from the parent quota.

Quota Coordinator Heartbeat Interval

alluxio.quota.worker.heartbeat.interval.ms controls the frequency of coordinator aggregating the current quota usage from workers in the cluster. Because the coordinator polls the quota usages from workers periodically, the quota usage it observes (also reflected in the bin/alluxio quota list command) will have a delay.

# Default value is 1s
alluxio.quota.worker.heartbeat.interval.ms=1s

Tracking Quota Usages in Prometheus Metrics

Other than the bin/alluxio quota list command, Alluxio also exposes the current quota usages in Prometheus metrics on the Coordinator process. The metrics are labeled with the Alluxio path of the quota definition. The cluster admin can track these Prometheus metrics in the Grafana dashboard and monitor the current quota usage without running the command.

# HELP alluxio_quota_size_capacity_bytes Capacity of the given quota scope as defined in the quota rules
# TYPE alluxio_quota_size_capacity_bytes gauge
alluxio_quota_size_capacity_bytes{dir="/s3"} 3.145728E8
alluxio_quota_size_capacity_bytes{dir="/s3/data"} 8.388608E7
alluxio_quota_size_capacity_bytes{dir="/local"} 1.048576E8
# HELP alluxio_quota_size_used_bytes Bytes used in the given quota scope
# TYPE alluxio_quota_size_used_bytes gauge
alluxio_quota_size_used_bytes{dir="/s3"} 3.145728E8
alluxio_quota_size_used_bytes{dir="/s3/data"} 0.0
alluxio_quota_size_used_bytes{dir="/local"} 0.0

Quota in Load Jobs

The Load command will load a directory or a large list of files specified in an index file into the cache. It is possible for a load operation to exceed a quota and cause undesired cache eviction. To avoid this scenario, a check can be enabled to reject the job if the load operation would exceed a quota.

This check will calculate and compare 3 pieces of information:

The total size of the files in the UFS
For those files, how much of them are already cached in Alluxio workers
The current availability of the quota definition

For example, if the index file specifies 100 files of total 100GB size, with 50GB of them are already cached in the workers and the quota currently only has 10GB availability left, Alluxio should reject this load job. The admin should manually free up at least 40GB from that quota before trying to submit the load job again.

Constraints and Configurations

Enable Quota Check Step on Load

By default, the quota check step in a Load job is disabled and none of the following constraints will be enforced. To enable the quota check step, set the following property:

# false by default
alluxio.dora.load.job.must.check.quota=true

Adding the --skip-quota-check will override the quota check enabled by the configuration property.

# Skip the quota check and start loading directly 
$ ./bin/alluxio job load --path s3://alluxio-test/data/ --submit --skip-quota-check

The quota check must be enabled for any of the following constraints to be enforced. If it is disabled by configuration or command line flag, no checks will occur.

Prevent Loading Files from Multiple Quota Definitions

The quota check will reject a load job if it involves files from multiple quota definitions. This applies to both loading a directory or loading a list of files specified in an index file. This constraint will always be enforced if quota check is enabled.

Enforce Quota Definition for All Load Jobs

The quota check can be configured to reject a load job if it tries to load a path that is not under a quota definition. This constraint can be used to ensure all load jobs are under the quota control. It can optionally be turned off from the set of quota checks by setting:

# true by default
alluxio.dora.load.job.without.quota.allowed=false

Prevent Concurrent Load Jobs Under One Quota Definition

Since the quota is only checked at the start of a load job, it is possible to exceed the quota if running concurrent load jobs under the same quota. To avoid this situation, the quota check can be configured to only allow one load job at a time to run within a path set with a quota. To opt into enforcing this restriction, set the property:

# false by default
alluxio.dora.load.job.quota.mutual.exclusive=true

Last updated 20 days ago