Cache Evicting by TTL

Overview and Use Cases

Alluxio supports defining TTL(time-to-live) rules on cache in different directories. Once a file’s cache exceeds its TTL, Alluxio will remove that cache automatically. The TTL is based on the time when the cache is loaded into Alluxio. So different parts or different replicas of a file may be evicted from Alluxio at different times.

A TTL rule just specify a maximal lifespan for the cache. It does not prevent the cache to be evicted due to other reasons, like the worker's cache capacity being full. You can find more details about when and how cache eviction is triggered in Cache Evicting.

The TTL feature can be used to automatically free up cache without human intervention. For example, if some cache is only useful for one day, the TTL can be set to 24 hours. The TTL feature can also be used to limit the lifespan of sensitive data in cache, in order to control risk and satisfy compliance requirements.

To enable this feature, set the configurations as below:

alluxio.ttl.policy.enabled=true
alluxio.ttl.eviction.check.interval=1h

The TTL rules are stored in ETCD. Alluxio workers will regularly poll ETCD to observe the defined TTL rules. Every 1 hour (controlled by alluxio.ttl.eviction.check.interval), a worker will scan all its cache and remove all cache whose TTL has expired.

Basic Operations

For complete description of the commands, please refer to the CLI reference.

Add a TTL rule

You can define a new TTL rule on a directory in Alluxio namespace.

$ bin/alluxio ttl add --path /s3/data/ --time 1h
Added alluxioPath=/s3/data/ and time=1h

We do not recommend defining a TTL length less than alluxio.ttl.eviction.check.interval, and a warning will be printed. The TTL rule will still be added. You may choose to either remove it or update it with a more reasonable TTL rule.

$ bin/alluxio ttl add --path /s3/small_interval/ --time 5s
Warning: You are setting TTL policy to 5s. This TTL is too small. Note that expired cache are scanned and evicted every 1h. Please consider making this TTL larger with `bin/alluxio ttl update` command.
Added alluxioPath=/s3/small_interval/ and time=5s

Alluxio also supports nested TTL rules. For a file, if multiple TTL rules apply to it, the most specific rule will take effect. For example, if you define multiple nested TTL rules, file /s3/data/team1/temporary/checkpoint will have a TTL of 1 hour.

$ bin/alluxio ttl add --path /s3/data/ --time 10h
Added alluxioPath=/s3/data/ and time=10h

$ bin/alluxio ttl add --path /s3/data/team1 --time 5h
Added alluxioPath=/s3/data/team1 and time=5h

$ bin/alluxio ttl add --path /s3/data/team1/temporary --time 1h
Added alluxioPath=/s3/data/team1/temporary and time=1h

There is no limit in how many levels of directory a TTL definition can have. There is also no limit in how many children a parent TTL rule can have. However, we do not recommend defining more than 100 TTL rules or more than 3 levels of nested TTL rules. That will introduce operational overhead to the cluster administrator.

Update a TTL rule

You can update a TTL rule after it was added:

$ bin/alluxio ttl update --path /s3/test_folder/ --time 30min
Updated alluxioPath=/s3/test_folder/ and time=30min

$ bin/alluxio ttl update --path /s3/test_folder/ --time 5s
Warning: You are setting TTL policy to 5s. This TTL is too small. Note that expired cache are scanned and evicted every 1h. Please consider making this TTL larger with `bin/alluxio ttl update` command.
Updated alluxioPath=/s3/test_folder/ and time=5s

Remove a TTL rule

This command is used to remove TTL policies from ETCD. Here are some examples:

$ bin/alluxio ttl remove --path /s3/test_folder
Removed TTL policy for alluxioPath=/s3/test_folder/

List all TTL rules

This command will list all the effective TTL policies:

$ bin/alluxio ttl list
Listing all TTL policies
The ttl policy table is empty.

$ bin/alluxio ttl list
Listing all TTL policies
/s3/test.csv/     TTL: 1 hour
/s3/test_2.csv/   TTL: 30 seconds

Force trigger TTL scanning

You can use this command to force all workers to trigger one round of cache TTL scanning immediately. If a worker is already in the middle of a regular scan, that worker will be skipped from scanning again.

$ bin/alluxio ttl trigger
Force triggering all workers to scan their cache with TTL rules.
Successfully scheduled on-demand TTL evictions.

# If two workers are already in the middle of a regular scan
$ bin/alluxio ttl trigger
Force triggering all workers to scan their cache with TTL rules.
2 workers are already scanning
Successfully scheduled on-demand TTL evictions.

Because there is system overhead in each full cache TTL scanning, we do not recommend heavily relying on this command. Instead, you should rely on the regular TTL scan at each worker to remove the expired cache. This command should only be used when the regular scan is not triggered correctly, or there is a need to quickly restore some space from cache.

Advanced configuration and operations

Configure the TTL scan interval

The default interval between each TTL scan is 1 hour. Because there is system overhead in each scan, we recommend using a large interval in your production environment, like:

alluxio.ttl.eviction.check.interval=24h

Observability

Currently, users can only monitor how much cache is evicted due to TTL through worker logs.