Alluxio
ProductsLanguageHome
  • Introduction
  • Overview
    • Architecture
    • Job Service
    • Quick Start Guide
    • FAQ
    • Use Cases
  • Core Services
    • Caching
    • Unified Namespace
  • Install Alluxio
    • Local Machine
    • Cluster
    • Cluster with HA
    • Docker
    • Software Requirements
  • Kubernetes
    • Deploy
    • Spark on Kubernetes
    • Metrics
  • Cloud Native
    • Alibaba Cloud ACK
    • AWS EMR
    • Tencent EMR
    • Google Dataproc
  • Compute Integration
    • Apache Spark
    • Apache Hadoop MapReduce
    • Apache Flink
    • Apache Hive
    • Presto on Iceberg (Experimental)
    • Presto
    • Trino
    • Tensorflow
  • Storage Integrations
    • Amazon AWS S3
    • HDFS
    • Azure Blob Store
    • Azure Data Lake Storage Gen2
    • Azure Data Lake Storage
    • Google Cloud Storage
    • Qiniu Kodo
    • COSN
    • CephObjectStorage
    • MinIO
    • NFS
    • Aliyun Object Storage Service
    • Ozone
    • Swift
    • WEB
    • CephFS
  • Security
  • Operations
    • Configuration Settings
    • User CLI
    • Admin CLI
    • Web UI
    • Journal Management
    • Metastore Management
    • Metrics
  • Administration
    • Troubleshooting
    • Basic Logging
    • Remote Logging
    • Performance Tuning
    • Scalability Tuning
    • StressBench (Experimental)
    • Upgrading
  • Solutions
  • Client APIs
    • Java API
    • S3 API
    • REST API
    • POSIX API
  • Contributor Resources
    • Building Alluxio From Source
    • Contribution Guide
    • Code Conventions
    • Documentation Conventions
    • Contributor Tools
  • Reference
    • List Of Configuration Properties
    • List of Metrics
  • REST API
    • Master REST API
    • Worker REST API
    • Proxy REST API
    • Job REST API
  • Javadoc
Powered by GitBook
On this page
  • REST API
  • Python
  • Install Python Client Library
  • Example Usage
  • Go
  • Install Go Client Library
  • Example Usage
  1. Client APIs

REST API

Last updated 6 months ago

While users should use for data I/O operations, admins can interact with Alluxio through REST API for actions not supported by S3 API. For example, mount and unmount operations.

REST API

For portability with other language, the ) is also accessible via an HTTP proxy in the form of a REST API. Alluxio's Python and Go clients rely on this REST API to talk to Alluxio.

The is generated as part of the Alluxio build and accessible through ${ALLUXIO_HOME}/core/server/proxy/target/miredot/index.html. The main difference between the REST API and the Alluxio Java API is in how streams are represented. While the Alluxio Java API can use in-memory streams, the REST API decouples the stream creation and access (see the create and open REST API methods and the streams resource endpoints for details).

The HTTP proxy is a standalone server that can be started using ${ALLUXIO_HOME}/bin/alluxio-start.sh proxy and stopped using ${ALLUXIO_HOME}/bin/alluxio-stop.sh proxy. By default, the REST API is available on port 39999.

There are performance implications of using the HTTP proxy. In particular, using the proxy requires an extra network hop to perform filesystem operations. For optimal performance, it is recommended to run the proxy server and an Alluxio worker on each compute node.

Python

Alluxio has a for interacting with Alluxio through its . The Python client exposes an API similar to the ). See the for detailed documentation about all available methods. See the on how to perform basic file system operations in Alluxio.

The Python client requires an Alluxio proxy that exposes the to function.

Install Python Client Library

$ pip install alluxio

Example Usage

The following program includes examples of how to create directory, download, upload, check existence for, and list status for files in Alluxio.

This example can also be found in the Python package's repository.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import json
import sys

import alluxio
from alluxio import option


def colorize(code):
    def _(text, bold=False):
        c = code
        if bold:
            c = '1;%s' % c
        return '\033[%sm%s\033[0m' % (c, text)
    return _


green = colorize('32')


def info(s):
    print(green(s))


def pretty_json(obj):
    return json.dumps(obj, indent=2)


py_test_root_dir = '/py-test-dir'
py_test_nested_dir = '/py-test-dir/nested'
py_test = py_test_nested_dir + '/py-test'
py_test_renamed = py_test_root_dir + '/py-test-renamed'

client = alluxio.Client('localhost', 39999)

info("creating directory %s" % py_test_nested_dir)
opt = option.CreateDirectory(recursive=True)
client.create_directory(py_test_nested_dir, opt)
info("done")

info("writing to %s" % py_test)
with client.open(py_test, 'w') as f:
    f.write('Alluxio works with Python!\n')
    with open(sys.argv[0]) as this_file:
        f.write(this_file)
info("done")

info("getting status of %s" % py_test)
stat = client.get_status(py_test)
print(pretty_json(stat.json()))
info("done")

info("renaming %s to %s" % (py_test, py_test_renamed))
client.rename(py_test, py_test_renamed)
info("done")

info("getting status of %s" % py_test_renamed)
stat = client.get_status(py_test_renamed)
print(pretty_json(stat.json()))
info("done")

info("reading %s" % py_test_renamed)
with client.open(py_test_renamed, 'r') as f:
    print(f.read())
info("done")

info("listing status of paths under /")
root_stats = client.list_status('/')
for stat in root_stats:
    print(pretty_json(stat.json()))
info("done")

info("deleting %s" % py_test_root_dir)
opt = option.Delete(recursive=True)
client.delete(py_test_root_dir, opt)
info("done")

info("asserting that %s is deleted" % py_test_root_dir)
assert not client.exists(py_test_root_dir)
info("done")

Go

Install Go Client Library

$ go get -d github.com/Alluxio/alluxio-go

Example Usage

If there is no Alluxio proxy running locally, replace "localhost" below with a hostname of a proxy.

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"strings"
	"time"

	alluxio "github.com/Alluxio/alluxio-go"
	"github.com/Alluxio/alluxio-go/option"
)

func write(fs *alluxio.Client, path, s string) error {
	id, err := fs.CreateFile(path, &option.CreateFile{})
	if err != nil {
		return err
	}
	defer fs.Close(id)
	_, err = fs.Write(id, strings.NewReader(s))
	return err
}

func read(fs *alluxio.Client, path string) (string, error) {
	id, err := fs.OpenFile(path, &option.OpenFile{})
	if err != nil {
		return "", err
	}
	defer fs.Close(id)
	r, err := fs.Read(id)
	if err != nil {
		return "", err
	}
	defer r.Close()
	content, err := ioutil.ReadAll(r)
	if err != nil {
		return "", err
	}
	return string(content), err
}

func main() {
	fs := alluxio.NewClient("localhost", 39999, 10*time.Second)
	path := "/test_path"
	exists, err := fs.Exists(path, &option.Exists{})
	if err != nil {
		log.Fatal(err)
	}
	if exists {
		if err := fs.Delete(path, &option.Delete{}); err != nil {
			log.Fatal(err)
		}
	}
	if err := write(fs, path, "Success"); err != nil {
		log.Fatal(err)
	}
	content, err := read(fs, path)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Result: %v\n", content)
}

Alluxio has a for interacting with Alluxio through its . The Go client exposes an API similar to the . See the for detailed documentation about all available methods. The godoc includes examples of how to download, upload, check existence for, and list status for files in Alluxio.

The Go client requires an Alluxio proxy that exposes the to function.

S3 API
Alluxio Java API
REST API documentation
Python Client
Alluxio Java API
doc
example
here
REST API
REST API
Go Client
Alluxio Java API
godoc
REST API
REST API