Home

Accessing Data

Alluxio provides multiple interfaces for applications to access data, ensuring compatibility with a wide range of existing tools and frameworks. It also offers powerful features to optimize performance and ensure high availability. This guide provides an overview of the primary data access methods and related features.

Core Data Access APIs

Alluxio offers several ways for applications and users to interact with the data it manages:

  • S3 API: Expose an S3-compatible endpoint, allowing applications built with AWS S3 SDKs (like Python's boto3 or the Java S3 client) to connect to Alluxio.

  • HDFS API: Connect to Alluxio as a HDFS compatible filesystem.

Performance Optimization

Alluxio includes several features designed to accelerate data I/O and metadata operations, ensuring your applications run at maximum speed.

  • Optimizing Read Performance: Learn how to use client-side prefetching and techniques for large file segmentation to maximize read throughput.

  • Optimizing Write Performance: Use a client-side or cluster-level write cache to accelerate write-intensive workloads like writing shuffle data, decoupling application performance from UFS latency.

  • Optimizing Metadata Performance: For directories containing millions of files, use the Index Service to create a distributed, scalable cache for directory listings, dramatically speeding up metadata operations like ls.

  • Controlling UFS Bandwidth : Configure a rate limit on reads from the UFS to prevent Alluxio from overwhelming the underlying storage system during cache-filling operations.

High Availability and Resiliency

Alluxio is designed to be resilient and ensure data is always available, even in the face of component or infrastructure failures.

  • UFS Fallback: If an Alluxio worker is unavailable, clients can automatically fall back to reading directly from the UFS, ensuring that read requests succeed without interruption.

  • Managing File Replication: Configure files to have multiple replicas across different Alluxio workers. If one worker becomes unavailable, clients can seamlessly fail over to another replica. This also boosts read performance for popular files by distributing the load.

  • Deploying in Multiple Availability Zones (Multi-AZ): Deploy Alluxio clusters across multiple availability zones (AZs). If an entire AZ goes down, clients can automatically fail over to an Alluxio cluster in another AZ, providing robust disaster recovery and uninterrupted service.

Last updated