Client Virtual Path Mapping

Overview

Alluxio supports mapping one path to another. All read/write and metadata operation requests will not be performed on the original path, but will instead be redirected to the mapped path.

The current client virtual path mapping is only applicable to Alluxio Fuse and the Alluxio CLI.

Enable Virtual Path Mapping

alluxio.user.virtual.path.mapping.enabled=true
alluxio.user.virtual.path.mapping.rule.file.path=/opt/alluxio/conf/path_mapping.json

The path_mapping.json file is a JSON file that describes the mapping relationships between paths. Here is a simple example: suppose you want to map /a/b/c/xxx to /b/c/d/xxx, you can configure it like this:

{
  "rules": [
    {
      "src": "^/a/b/c/(.*)$",
      "dst": "/b/c/d/{{ var1 }}"
    }
  ]
}

Here, src is a regular expression matching the original path. Parentheses are used to capture values, which can then be used in the dst using var placeholders:

• var1 refers to the first captured group,

• var2 to the second, and so on.

• varN represents the N-th captured group.

As another example, suppose we want the following path mappings:

/foo/bar/[a-b].*  ------> /foo1/bar1/A/xxx
/foo/bar/[e-g].*  ------> /foo1/bar1/B/xxx
/foo/bar/[\d].*   ------> /foo1/bar1/C/xxx
/foo/bar/[acd].*  ------> /foo1/bar1/D/xxx

The configuration file can be written as follows. Note: when writing the \ character in a JSON file, it must be escaped as \\ because \ is a JSON escape character:

{
  "rules": [
    {
      "src": "^/foo/bar/([a-b].*)$",
      "dst": "/foo1/bar1/A/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([e-g].*)$",
      "dst": "/foo1/bar1/B/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([\\d].*)$",
      "dst": "/foo1/bar1/C/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([acd].*)$",
      "dst": "/foo1/bar1/D/{{ var1 }}"
    }
  ]
}

Note: Mapping rules are matched from top to bottom. Only the first matching rule takes effect. If no rule matches a given path, the original path is used as the mapped path.

Path Mapping Validation Tool

A command-line tool is provided to verify that your path mapping rules behave as expected. It is strongly recommended to use this tool before deploying:

bin/alluxio fs pathmapping /foo/bar/a.txt

-----------------------------------------------------------------
Input:  /foo/bar/a.txt
Output: /foo1/bar1/A/1.txt
Virtual Directory: false
-----------------------------------------------------------------

The Output field shows the mapped path. If no rule matches the input path, the value of Output will be null. The meaning of the Virtual Directory field is explained in the following section.

Fuse Path Mapping (Important)

The virtual path mapping configuration uses Alluxio's cluster paths. When applied to local paths through Alluxio Fuse, the Fuse mount path must be taken into account.

Take the following rule as an example:

{
  "rules": [
    {
      "src": "^/a/b/(.*)$",
      "dst": "/a1/b1/{{ var1 }}"
    }
  ]
}

The following commands are equivalent, since /a/b/c is mapped to /a1/b1/c:

bin/alluxio fs ls /a/b/c
bin/alluxio fs ls /a1/b1/c

Assuming Alluxio Fuse is mounted locally at /mnt/alluxio/alluxio-fuse, then the above rule effectively maps /mnt/alluxio/alluxio-fuse/a/b/c to /mnt/alluxio/alluxio-fuse/a1/b1/c. Therefore, the following commands are equivalent in this context:

ls /mnt/alluxio/alluxio-fuse/a/b/c
ls /mnt/alluxio/alluxio-fuse/a1/b1/c

Virtual Directories

When accessing a file via Alluxio Fuse, Alluxio will check if the file’s parent directories exist. If a parent directory is missing, an error is thrown.

Consider this rule:

{
  "rules": [
    {
      "src": "^/a/b/(.*)$",
      "dst": "/a1/b1/{{ var1 }}"
    }
  ]
}

When reading /a/b/c, Alluxio Fuse will check whether the parent path /a/b exists. If /a/b is missing, an error will occur. This check is unrelated to whether the mapped path /a1/b1 exists. To avoid such errors, you can configure virtual directories:

{
  "rules": [
    {
      "src": "^/a/b/(.*)$",
      "dst": "/a1/b1/{{ var1 }}"
    }
  ],
  "virtualDirectories": [
    "/a", "/a/b"
  ]
}

The virtualDirectories field can contain multiple regex expressions, which are matched in order. Paths matching virtualDirectories are always treated as existing directories.

You can validate virtual directories using the command-line tool:

bin/alluxio fs pathmapping /a
-----------------------------------------------------------------
Input:  /a
Output: null
Virtual Directory: true
-----------------------------------------------------------------

The Virtual Directory field in the output indicates whether the path matches a virtual directory.

Limitations

Number of Rules

Virtual path mapping is not suitable for scenarios with a large number of rules, as matching each rule with a regular expression can affect performance.

ls Limitations

When listing a directory using ls, not all matched sub-paths will be visible.

For example, with the following mapping rules:

{
  "rules": [
    {
      "src": "^/foo/bar/([a-b].*)$",
      "dst": "/foo1/bar1/A/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([e-g].*)$",
      "dst": "/foo1/bar1/B/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([\\d].*)$",
      "dst": "/foo1/bar1/C/{{ var1 }}"
    },
    {
      "src": "^/foo/bar/([acd].*)$",
      "dst": "/foo1/bar1/D/{{ var1 }}"
    }
  ]
}

Running ls /foo/bar will not show:

/foo1/bar1/A
/foo1/bar1/B
/foo1/bar1/C
/foo1/bar1/D

However, you can still access content directly under /foo/bar, such as:

cat /foo1/bar1/A/1.txt