Alluxio
ProductsLanguageHome
  • Introduction
  • Overview
    • Architecture
    • Job Service
    • Quick Start Guide
    • FAQ
    • Use Cases
  • Core Services
    • Caching
    • Unified Namespace
  • Install Alluxio
    • Local Machine
    • Cluster
    • Cluster with HA
    • Docker
    • Software Requirements
  • Kubernetes
    • Deploy
    • Spark on Kubernetes
    • Metrics
  • Cloud Native
    • Alibaba Cloud ACK
    • AWS EMR
    • Tencent EMR
    • Google Dataproc
  • Compute Integration
    • Apache Spark
    • Apache Hadoop MapReduce
    • Apache Flink
    • Apache Hive
    • Presto on Iceberg (Experimental)
    • Presto
    • Trino
    • Tensorflow
  • Storage Integrations
    • Amazon AWS S3
    • HDFS
    • Azure Blob Store
    • Azure Data Lake Storage Gen2
    • Azure Data Lake Storage
    • Google Cloud Storage
    • Qiniu Kodo
    • COSN
    • CephObjectStorage
    • MinIO
    • NFS
    • Aliyun Object Storage Service
    • Ozone
    • Swift
    • WEB
    • CephFS
  • Security
  • Operations
    • Configuration Settings
    • User CLI
    • Admin CLI
    • Web UI
    • Journal Management
    • Metastore Management
    • Metrics
  • Administration
    • Troubleshooting
    • Basic Logging
    • Remote Logging
    • Performance Tuning
    • Scalability Tuning
    • StressBench (Experimental)
    • Upgrading
  • Solutions
  • Client APIs
    • Java API
    • S3 API
    • REST API
    • POSIX API
  • Contributor Resources
    • Building Alluxio From Source
    • Contribution Guide
    • Code Conventions
    • Documentation Conventions
    • Contributor Tools
  • Reference
    • List Of Configuration Properties
    • List of Metrics
  • REST API
    • Master REST API
    • Worker REST API
    • Proxy REST API
    • Job REST API
  • Javadoc
Powered by GitBook
On this page
  • Architecture Overview
  • Different Types of Jobs
  • Migrate
  • Persist
  • Evict
  • Move
  • Replicate
  • Inspection Commands
  • fsadmin report jobservice
  • job ls
  • job stat -v <job_id>
  1. Overview

Job Service

Last updated 6 months ago

Architecture Overview

The Alluxio Job Service is a task scheduling framework responsible for assigning a number of different types of operations to Job Workers.

The Master is responsible for distributing a job into smaller tasks for the Job Workers to execute and managing the status of the job's completion.

The Job Workers queues tasks from the Job Master and manages a configurable fixed threadpool (alluxio.job.worker.threadpool.size) to complete these tasks.

Different Types of Jobs

Migrate

Migrate Job is used as part of the fs distributedCp CLI command and is responsible for copying with a specified .

Persist

Persist Job is used both indirectly as part of the fs persist CLI command and background persisting when writing to Alluxio with ASYNC_THROUGH .

The job is responsible for persisting a file in Alluxio to a particular ufs path.

Evict

Evict Job is indirectly used by fs free CLI command and replication background process.

The job is responsible for evicting a specified number of replicated blocks from Alluxio.

Move

Move Job is used by the replication background process to move blocks from one worker to another.

Replicate

Replicate job is used by the replication background process to replicate blocks from a worker to a specified number of other workers.

Inspection Commands

The job service provides a number of inspection commands.

fsadmin report jobservice

fsadmin report jobservice will report a summary of the job service.

$ ./bin/alluxio fsadmin report jobservice
Worker: MigrationTest-workers-2  Task Pool Size: 10     Unfinished Tasks: 1303   Active Tasks: 10     Load Avg: 1.08, 0.64, 0.27
Worker: MigrationTest-workers-3  Task Pool Size: 10     Unfinished Tasks: 1766   Active Tasks: 10     Load Avg: 1.02, 0.48, 0.21
Worker: MigrationTest-workers-1  Task Pool Size: 10     Unfinished Tasks: 1808   Active Tasks: 10     Load Avg: 0.73, 0.5, 0.23

Status: CREATED   Count: 4877
Status: CANCELED  Count: 0
Status: FAILED    Count: 1
Status: RUNNING   Count: 0
Status: COMPLETED Count: 8124

10 Most Recently Modified Jobs:
Timestamp: 10-28-2020 22:02:34:001       Id: 1603922371976       Name: Persist             Status: COMPLETED
Timestamp: 10-28-2020 22:02:34:001       Id: 1603922371982       Name: Persist             Status: COMPLETED
(only a subset of the results is shown)

10 Most Recently Failed Jobs:
Timestamp: 10-24-2019 17:15:22:946       Id: 1603922372008       Name: Persist             Status: FAILED

10 Longest Running Jobs:

job ls

job ls will list jobs that are running or have run on the job service.

$ ./bin/alluxio job ls
1613673433925   Persist    COMPLETED
1613673433926   Persist    COMPLETED
1613673433927   Persist    COMPLETED
1613673433928   Persist    COMPLETED
1613673433929   Persist    COMPLETED

job stat -v <job_id>

job stat -v <job_id> will list detailed information about a job. (Adding -v includes information about the individual tasks on the workers)

bin/alluxio job stat -v 1613673433929
ID: 1613673433929
Name: Persist
Description: PersistConfig{filePath=/test5/lib/alluxio-underfs-cosn-2.5.0-SNAPSHOT.jar, mountId=1, overwrite=false, ufsPath=...
Status: COMPLETED
Task 0
	Worker: 192.168.42.71
	Status: COMPLETED
WriteType
WriteType