Ctrlk
Home
For the complete documentation index, see llms.txt. This page is also available as Markdown.

RDMA Networking

Alluxio supports several high-speed network technologies commonly deployed in AI and HPC clusters. This page covers configuration and performance guidance for each supported option.

Technology
Status
Use Case

Native RDMA over InfiniBand or RoCE

✅ Supported

Zero-copy data transfer, bypasses kernel network stack entirely

IPoIB (IP over InfiniBand)

✅ Supported

Standard TCP/IP over IB hardware

AWS EFA

Not supported

iWARP

Not supported

Native RDMA

Note: This feature is experimental since AI-3.9-16.0.0.

Overview

Native RDMA bypasses the kernel network stack and enables zero-copy data transfer between Alluxio workers and clients. Compared to IPoIB (which carries TCP/IP over InfiniBand hardware), it delivers significantly lower latency and higher throughput for cached reads.

Scope

  • Read I/O only; write paths continue to use TCP.

  • InfiniBand and RoCE only; AWS EFA and iWARP are not supported.

Recommended for

  • Clusters where workers and clients are connected via InfiniBand or RoCE

  • Latency-sensitive workloads (e.g., model loading, inference serving)

  • Maximizing Alluxio cache read throughput

Prerequisites

Hardware

  • InfiniBand or RoCE-capable NICs (e.g., Mellanox/NVIDIA ConnectX-4 or later)

  • InfiniBand fabric or RoCE-enabled Ethernet switches

  • RDMA NICs installed on all worker nodes and all client/FUSE nodes in the data path

  • AWS EFA and iWARP adapters are not supported for native RDMA

Software

  • Linux OS with RDMA kernel modules loaded (kernel 5.0+ recommended)

  • MLNX OFED ≥ 25.10 (or DOCA-OFED) or upstream rdma-core ≥ 35

  • OpenUCX 1.20 or higher installed on all nodes, with an identical UCX version across all worker and client nodes

  • JUCX (Java bindings for UCX) — bundled with Alluxio

Environment Verification

Before enabling Alluxio native RDMA, verify that the RDMA environment is correctly configured.

Verify RDMA devices and port status

Expected output (key fields):

Verify:

  • state is PORT_ACTIVE

  • link_layer is InfiniBand (native IB) or Ethernet (IB NIC in Ethernet mode using RoCE)

If ibv_devinfo is not found, install rdma-core or MLNX_OFED drivers.

Install and verify UCX

Alluxio uses OpenUCX for native RDMA transport. Install UCX 1.20 or later on all worker and client nodes.

Via package manager (Ubuntu/Debian):

Via package manager (RHEL/CentOS):

Or build from source for the latest version:

Configure flags explained:

Flag
Purpose

--with-verbs

Enable InfiniBand Verbs transport (required for RDMA data transfer)

--with-rdmacm

Enable RDMA Connection Manager (required for connection establishment)

Verify installation:

Expected output: UCX version 1.20 or later.

Verify that RDMA transports are available:

Expected output: one or more transport devices listed. If none appear, the RDMA development libraries were not found during UCX build — reinstall libibverbs-dev and librdmacm-dev, then rebuild UCX.

For more installation options, see the OpenUCX documentation.

Verify bRPC environment (recommended)

bRPC provides RDMA-accelerated metadata RPCs and is recommended for best performance in RDMA environments. Verify the bRPC native library is available:

Expected output: one or more libbrpc*.so files. If not found, install the bRPC native library separately. Native RDMA data transfer (UCX) works without bRPC, but metadata RPCs will fall back to gRPC over TCP.

Quick Start

This section provides the minimum steps to enable native RDMA. See Configuration for advanced tuning.

Add the following to alluxio-site.properties on all worker and client nodes:

Restart all worker and client processes. Verify the worker RDMA service is running:

Important: When alluxio.network.rdma.data.enabled=true, clients use RDMA exclusively for data reads — there is no automatic fallback to TCP. If a worker's RDMA port is unreachable, the read operation fails (or retries on other replica workers if available).

Configuration

Data Acceleration

Property
Default
Description

alluxio.network.rdma.data.enabled

false

Enables RDMA data transfer on workers and clients

alluxio.network.rdma.data.bind.device

(unset)

RDMA Verbs device name(s) for the data plane (e.g., mlx5_0:1). Device names are obtained via ibv_devinfo (format: device_name:port_number). Multiple devices can be comma-separated (e.g., mlx5_0:1,mlx5_1:1). If a single device is specified, Alluxio auto-resolves the RDMA listener address from that NIC. If unset, UCX auto-selects all available RDMA devices.

alluxio.network.rdma.data.bind.host

0.0.0.0

Bind host/IP for the worker RDMA data listener. This is mainly used when bind address auto-resolution from alluxio.network.rdma.data.bind.device is not desired or when multiple RDMA devices are configured.

alluxio.network.rdma.data.hostname

(unset)

Hostname or IP that clients use to connect to the worker RDMA data service. If unset, the worker's general hostname is used.

alluxio.network.rdma.data.port

59999

Port for the worker RDMA data service.

In most single-NIC deployments, the defaults work without extra address configuration. For multi-NIC environments, set alluxio.network.rdma.data.hostname to the client-reachable IP, use alluxio.network.rdma.data.bind.device to pin the RDMA NIC, and override alluxio.network.rdma.data.bind.host if you do not want the bind address inferred from a single RDMA device.

Buffer Pool

Native RDMA data transfer uses a registered memory buffer pool. Tune these settings when workloads need higher RDMA concurrency, larger prefetch capacity, or lower first-read latency.

Property
Default
Description

alluxio.network.rdma.data.buffer.pool.initial.size

256MB

Initial size budget of the RDMA registered buffer pool at startup.

alluxio.network.rdma.data.buffer.pool.prewarm.size

256MB

Amount of RDMA registered buffer pool memory to pre-warm in the 4MB chunk bucket at startup. This must not exceed alluxio.network.rdma.data.buffer.pool.initial.size.

alluxio.network.rdma.data.buffer.pool.max.size

8GB

Maximum size of the RDMA registered buffer pool. The pool does not expand beyond this limit, and RDMA prefetching stops when the pool is full.

RDMA Prefetcher

The RDMA prefetcher reuses the legacy async prefetcher configuration keys. These properties have the same sequential-read semantics as the standard client-side async prefetcher, but RDMA uses them to control RDMA chunk prefetching instead of TCP-based async prefetch reads.

For the general async prefetch behavior, see File Reading.

The following properties are the recommended RDMA prefetcher tunables. Other inherited async prefetcher properties are not recommended for customization unless advised by Alluxio support.

Property
Default
RDMA behavior

alluxio.user.position.reader.streaming.async.prefetch.max.part.number

8

Maximum number of RDMA chunks to prefetch ahead for each stream. This is shared with the legacy async prefetcher.

alluxio.user.position.reader.streaming.async.prefetch.file.length.threshold

0

If a file length is less than or equal to this value, the RDMA prefetcher attempts to prefetch the whole file immediately. Values less than or equal to 0 disable this small-file behavior.

Recommended tuning:

  • For large-file reads, if client and worker memory resources are sufficient, set alluxio.user.position.reader.streaming.async.prefetch.max.part.number=16 and increase alluxio.network.rdma.data.buffer.pool.max.size=16GB accordingly.

  • For small-file reads through FUSE, enable alluxio.fuse.open.read.status.cache.enabled=true to reduce repeated read-only open metadata RPCs, and set alluxio.user.position.reader.streaming.async.prefetch.file.length.threshold to the largest file size you want RDMA to prefetch immediately.

Metadata Acceleration

Property
Default
Description

alluxio.network.rdma.metadata.enabled

false

Enables the experimental worker RDMA metadata path. When disabled, workers do not start the metadata transport server and clients use the existing RPC path.

alluxio.network.rdma.metadata.port

29995

Port for the worker RDMA metadata service.

alluxio.network.rdma.metadata.device

(unset)

RDMA device used by the metadata service. When unset, the metadata transport chooses its default active device.

alluxio.network.rdma.metadata.ib.port

1

InfiniBand port used by the metadata service.

alluxio.network.rdma.metadata.gid.index

-1

GID index used by the metadata service. Negative means the metadata transport chooses its default GID.

The worker RDMA metadata service uses the worker host/bind host for endpoint addressing. alluxio.network.rdma.metadata.port controls the service port, while alluxio.network.rdma.metadata.* controls which RDMA device/port/GID the native metadata transport uses.

Diagnostics

Property
Default
Description

alluxio.network.rdma.data.trace.enabled

false

Enable detailed per-operation RDMA data transport trace statistics in logs.

alluxio.network.rdma.data.trace.print.interval.ms

20s

Interval for printing RDMA data transport trace statistics when tracing is enabled.

Benchmark

Alluxio provides an RDMA benchmark CLI command to verify connectivity and measure raw performance before running production workloads.

Command Help

Available flags:

Flag
Default
Description

--mode

(auto-inferred)

Benchmark mode: server or client (auto-inferred from --remote-ip)

--local-ip

(auto-detected)

Local IP address

--local-port

20600

Local port

--delay-us

0

Simulated server-side processing delay in microseconds

--remote-ip

-

Remote IP address (client mode only, required)

--remote-port

20600

Remote port (client mode only)

--case

all

Test case: basic, latency, stress, stress-write, stress-read, all

--duration

10

Benchmark duration in seconds

--numjobs

-1

Number of concurrent threads (-1 for auto-adjustment based on payload size)

--iodepth

-1

Max in-flight operations per thread (-1 for auto-adjustment based on payload size)

Start Server

Start the benchmark server on one node:

Example output:

Run Latency Test

Start the client on another RDMA-connected node and run a latency test:

Example output:

Run Stress Read Test

Run a stress read test with custom concurrency parameters:

Example output:

Troubleshooting

No RDMA devices found

If the directory is empty:

  • Verify the NIC is physically installed: lspci | grep -i mellanox

  • Verify RDMA kernel modules are loaded: lsmod | grep ib_core

  • Install or reinstall the MLNX_OFED driver

RDMA port not in ACTIVE state

  • Check physical cable connections

  • Verify switch configuration

  • Run ibstat for detailed adapter and port status

UCX initialization failure

  • Verify UCX is installed: ucx_info -v

  • Check if UCX_NET_DEVICES is set to the correct device (e.g., mlx5_0:1)

  • Ensure LD_LIBRARY_PATH includes the UCX library path

RDMA operation timeouts

  • Increase alluxio.network.rdma.data.operation.timeout.ms (e.g., 60s)

  • Check network health via ibstat, verify port state is ACTIVE

  • Verify the worker RDMA port is reachable from client nodes

Buffer pool exhaustion

  • If frequent allocation failures occur, increase alluxio.network.rdma.data.buffer.pool.max.size

  • If first-read latency spikes are observed, increase alluxio.network.rdma.data.buffer.pool.initial.size

RDMA port busy after restart

  • The RDMA connection manager port may remain busy for up to 10 seconds after process exit. The server retries automatically with 1-second backoff.

  • If the port remains stuck, verify no other process is using it:

Client RDMA reads fail

  • RDMA does not fall back to TCP. Ensure all target workers have alluxio.network.rdma.data.enabled=true and the RDMA port is reachable.

  • Check client logs for RDMA connection errors.

IPoIB

Overview

InfiniBand (IB) is a high-bandwidth, low-latency interconnect commonly deployed in AI training clusters. Alluxio supports IP over InfiniBand (IPoIB), which runs the standard TCP/IP stack over IB hardware. Because Alluxio communicates over standard TCP/IP sockets, no code changes or special drivers are required — you only need to load the IPoIB kernel module and bind Alluxio services to the IB network interface.

Applies to: NICs configured with InfiniBand link layer (verified via ibstat | grep "Link layer"). If your ConnectX adapter is running in Ethernet link layer mode, it operates as a standard high-speed Ethernet NIC — Alluxio works with it natively with no IPoIB configuration needed.

IPoIB vs. Native RDMA

IPoIB
Native RDMA

Protocol

TCP/IP over IB hardware

Bypass kernel, zero-copy direct memory access

Alluxio support

✅ Fully supported

✅ Fully supported

Configuration

Bind to IB network interface

See Configuration

Typical throughput

100–400 Gbps (hardware-dependent)

Lower latency, similar peak bandwidth

Prerequisites

Hardware

  • Mellanox/NVIDIA ConnectX-4 or later network adapter

  • InfiniBand switch fabric

Software

Load the IPoIB kernel module and verify that the IB drivers and interfaces are active:

MTU Configuration

IPoIB operates in one of two transport modes that determine the maximum supported MTU:

Mode
Max MTU
Typical environments

Datagram (UD)

2,044 bytes

Cloud-managed IB (Azure HPC, AWS EFA)

Connected (RC)

65,520 bytes

On-premises InfiniBand fabrics

Check the current mode before setting MTU:

If mode is datagram (common on cloud IPoIB), the hardware limit is 2,044 bytes. Setting MTU to 9000 will fail with RTNETLINK answers: Invalid argument — this is expected, not an error. Alluxio works correctly at MTU 2,044.

If mode is connected (typical on-premises), set MTU to 9000 for maximum throughput:

To persist the MTU setting across reboots, add it to your network configuration (e.g., /etc/network/interfaces or a systemd-networkd unit file).

Binding Alluxio to the IB Interface

The hot data path in Alluxio runs between workers and FUSE / client nodes — this is where IB bandwidth matters. The coordinator handles background tasks (metadata operations, background jobs) and is not on the data-serving critical path, so it does not need to run on IB-equipped hardware.

For general NIC binding configuration, see Cluster Management. The steps below extend that guidance specifically for IPoIB deployments.

IPoIB can be exposed to pods through:

  • NVIDIA Network Operator: Automates MLNX_OFED driver deployment and SR-IOV device plugin configuration

  • Multus CNI: Attaches a secondary IB network interface to Alluxio pods

  • SR-IOV Device Plugin: Exposes IB Virtual Functions (VFs) as pod resources

Refer to NVIDIA Network Operator documentation and Multus CNI for setup instructions. Once the IB interface is available inside the pod, apply the same alluxio-site.properties settings from the Bare-Metal tab.

Worker Configuration

Add the following to alluxio-site.properties on each worker node. Replace ib0 with your actual IB interface name (check with ip addr show):

Verify after starting the worker:

FUSE / Client Configuration

For nodes running Alluxio FUSE or direct client access, bind the data channel to the IB interface:

For FUSE mount options and prerequisites (including allow_other configuration), see POSIX API (FUSE).

Coordinator Configuration

The coordinator does not need to run on IB-equipped hardware. Set alluxio.coordinator.hostname to the coordinator node's reachable IP address (typically its Ethernet interface):

Verify End-to-End Connectivity

After starting all services, confirm that worker–client data traffic flows over the IB interface:

Reference Performance

The following results are from an example test environment using IPoIB with Alluxio running on bare metal.

Test Environment

Parameter
Value

Network

2 × 200 Gbps IPoIB (bonded), measured throughput: 360 Gbps

NIC

Mellanox ConnectX-7 (IB link layer, 200 Gbps)

Cache disk

RAID0, 2 × NVMe, read/write: ~12 GB/s

UFS

Object storage via 100 Gbps dedicated line

Deployment

Bare metal, FUSE and worker co-located

Network Layer (iperf3)

Configuration
Measured Throughput

Single IB port

180 Gb/s

Bonded (2 × 200 Gbps IPoIB)

360 Gbps

Alluxio Read Throughput (Hot Read, Large Files, 32 Concurrent)

Configuration
Sequential Read

1 FUSE + 1 worker, 1 × NVMe

6.3 GB/s

1 FUSE + 1 worker, RAID0 2 × NVMe

12.5 GB/s

3 FUSE + 3 workers, RAID0 2 × NVMe

36.6 GB/s

Observation: With 3 workers and RAID0 NVMe cache, Alluxio hot read throughput approaches the raw disk bandwidth ceiling (~36 GB/s vs. 36 GB/s theoretical RAID0 maximum), confirming that the IPoIB network is not the bottleneck at this scale.

Troubleshooting

Worker not binding to IB interface

  • Run ip addr show ib0 to confirm the interface has an IP address assigned.

  • Verify that alluxio.worker.rpc.bind.device matches the exact interface name (case-sensitive).

  • Check alluxio-worker.log for bind errors.

Workers serve data over Ethernet instead of IB

  • Verify that alluxio.worker.data.bind.device=ib0 is set on each worker node and that the worker process was restarted after the change.

  • Verify that alluxio.user.network.data.bind.device=ib0 is set on the FUSE / client node.

ip link set ib0 mtu 9000 fails with RTNETLINK answers: Invalid argument

  • Your IPoIB interface is in datagram mode, which caps MTU at 2,044 bytes. This is common on cloud-managed InfiniBand (Azure HPC, AWS EFA). Alluxio works correctly at the default MTU — no action needed. See MTU Configuration.

Low throughput despite IPoIB

  • Run iperf3 -c <other node ib0 IP> between nodes to establish a network-layer baseline.

  • Check cat /sys/class/net/ib0/mode — datagram mode (MTU 2,044) will limit peak throughput compared to connected mode (MTU 9,000).

  • Confirm all services are communicating over the IB interface: ss -tnp | grep <ib0 IP>.

IB interface missing after reboot

  • MTU and bonding settings may not have been persisted. Add them to the system network configuration.

  • Verify MLNX_OFED drivers load on boot: lsmod | grep ib_core

Last updated