Contributor Tools
Last updated
Last updated
We recommend using IntelliJ IDEA to contribute to Alluxio. Eclipse can also be used. Instructions for setting up both IDEs can be found below.
To use IntelliJ IDEA to contribute to Alluxio, simply open IntelliJ and select "Import existing project". Then select the "Maven" project type from the IntelliJ dialog. IntelliJ's default configuration works without any modifications.
After successfully importing your local Alluxio repo into IntelliJ, you may need to add the Maven profile 'developer' in order to avoid import errors.
You can do this by going to
View > Tool Windows > Maven
In the Maven panel, find the "developer" profile and check the box next to it in the "Profiles" list.
Some source files in Alluxio are generated from templates or compiled from other languages.
gRPC and ProtoBuf definitions are compiled into Java source files. Alluxio 2.2 moved generated gRPC proto source files into core/transport/target/generated-sources/protobuf/
.
Compile time project constants are defined in core/common/src/main/java-templates/
and compiled to core/common/target/generated-sources/java-templates/
.
You will need to mark these directories as "Generated Sources Root" for IntelliJ to resolve the source files. Alternatively, you can let IntelliJ generate them and mark the directories automatically by running "Generate Sources and Update Folders for All Projects". You can find the button to trigger the generation at the top of the Maven panel, or you can search for this action from the Navigate > Search Everywhere
dialog.
See also and .
Start a single master Alluxio cluster
Run dev/intellij/install-runconfig.sh
Restart IntelliJ IDEA
Edit conf/alluxio-site.properties
to contain these configurations
Edit conf/log4j.properties
to print log in console Replace the log4j.rootLogger
configuration with
and add the following configurations
Format the Alluxio master by running bin/alluxio formatMasters
In Intellij, start Alluxio master process by selecting Run > Run > AlluxioMaster
In Intellij, start Alluxio job master process by selecting Run > Run > AlluxioJobMaster
Prepare the RamFS and format the Alluxio Worker with bin/alluxio-mount.sh SudoMount && bin/alluxio formatWorker
In Intellij, start Alluxio worker process by selecting Run > Run > AlluxioWorker
In Intellij, start Alluxio job worker process by selecting Run > Run > AlluxioJobWorker
Start a High Availability (HA) Alluxio cluster
Create journal directories for the masters
These directories are defined in the run configurations, i.e. alluxio/dev/intellij/runConfigurations/AlluxioMaster_0.xml
.
Note: If the journal folders exist, and you want to apply a new HA cluster, you should clear files in the journal folders first.
Run dev/intellij/install-runconfig.sh
Restart IntelliJ IDEA
Edit conf/alluxio-site.properties
to contain these configurations
The ports are defined in the run configurations.
In Intellij, start the Alluxio master processes by selecting Run > Run > AlluxioMaster-0
, Run > Run > AlluxioMaster-1
, and Run > Run > AlluxioMaster-2
Prepare the RamFS and format the Alluxio Worker with bin/alluxio-mount.sh SudoMount && bin/alluxio formatWorker
In Intellij, start the Alluxio worker process by selecting Run > Run > AlluxioWorker
In Intellij, start the Alluxio job master process by selecting Run > Run > AlluxioJobMaster
In Intellij, start the Alluxio job worker process by selecting Run > Run > AlluxioJobWorker
Verify the HA Alluxio cluster is up, by running bin/alluxio fsadmin journal quorum info -domain MASTER
, and you will see output like this:
You can also start a High Availability (HA) Job Master process on this basis.
Stop the Alluxio job master and job worker processes from steps 8 and 9 if they are running.
Edit conf/alluxio-site.properties
and add these configurations
In Intellij, start the Alluxio job master processes by selecting Run > Run > AlluxioJobMaster-0
, Run > Run > AlluxioJobMaster-1
, and Run > Run > AlluxioJobMaster-2
In Intellij, start the Alluxio job worker process by selecting Run > Run > AlluxioJobWorker
Verify the HA JobMaster cluster is up, by running bin/alluxio fsadmin journal quorum info -domain JOB_MASTER
, and you will see output like this:
Start an AlluxioFuse process
In Intellij, start AlluxioFuse process by selecting Run > Run > AlluxioFuse
. This creates a FUSE mount point at /tmp/alluxio-fuse
.
Verify the FUSE filesystem is working by running these commands:
You should be able to see the file is created and listed by both ls
commands.
Starting multiple processes in IntelliJ at once
IntelliJ is capable of creating groups of processes that all be launched simultaneously. To do so go to Run > Edit Configurations > + > Compound
. From there you can create a group of processes that can be launched together using a single Run > Run >
command. This can be useful when launching clusters from IntelliJ.
Import the folder into Eclipse. You may also have to add the classpath variable M2_REPO
by running:
Note: Alluxio 2.2 moved generated gRPC proto source files into
alluxio/core/transport/target/generated-sources/protobuf/
. You will need to mark the directory as a source folder for Eclipse to resolve the source files.
Before pushing changes or submitting pull requests, we recommend running various maven targets on your local machine to make sure your changes do not break existing behavior.
For these maven commands we'll assume that your command terminal is located in the root directory of your local copy of the Alluxio repository.
To make sure your code follows our style conventions you may run. Note that this is run any time you run targets such as compile
, install
, or test
.
To simply compile the code you can run the following command:
This will not execute any unit tests but will execute maven plugins such as checkstyle
and spotbugs
.
To speed up compilation you may use the following command:
This command will skip many of our checks that are in place to help keep our code neat. We recommend running all checks before committing.
-DskipTests
skips running unit and integration tests
-Dmaven.javadoc.skip
skips javadoc generation
-Dfindbugs.skip
skips findbugs execution
-Dcheckstyle.skip
skips code-style checking
-Dlicense.skip
skips checking files for license headers
-pl '!webui'
skips building the Alluxio UI module. If this module isn't compiled then the UI cannot be accessed locally.
You may replace the compile
target in the above command with any other valid maven target to skip checks as well. The targets compile
, verify
, and install
are typically the most useful.
If you want to test your changes with a compiled version of the repository, you may generate the jars with the Maven install
target. The first time Maven executes it will likely need to download many dependencies. Please be patient as the first build may take a while.
This will use the local filesystem as the under storage.
You can execute the maven test
command targeting the desired submodule directory. For example, to run tests for HDFS UFS module you would run
The above unit tests will create a simulated HDFS service with a specific version. To run more comprehensive tests on HDFS under storage using a real and running HDFS deployment:
To have the logs output to STDOUT, append the following arguments to the mvn
command
./bin/alluxio
Some commands have different prerequisites.
.
Start a or a in Intellij.
Before submitting the pull-request, run the latest code against the Maven plugin to verify no new warnings are introduced.
-T 2C
runs maven with
After the install target executes, you can follow the instructions at to start a local cluster.
The FUSE tests are ignored if the libfuse
library is missing. To run those tests, please install the libraries referenced in .
Alluxio uses 1.37.0 for RPC communication between clients and servers. The .proto
files defined in core/transport/src/grpc/
are used to auto-generate Java code for calling the RPCs on clients and implementing the RPCs on servers. To regenerate Java code after changing a gRPC definition, you must rebuild alluxio-core-transport
module with 'generate'
maven profile.
Alluxio uses 3.19 to read and write journal entries. The .proto
files defined in core/transport/src/proto/
are used to auto-generate Java definitions for the protocol buffer messages.
To change one of these messages, first read about to make sure your change will not break backwards compatibility. To regenerate Java code after changing a definition, you must rebuild alluxio-core-transport
module with the 'generate'
maven profile.
Please refer to for all available commands.
All commands except bootstrapConf
, killAll
, copyDir
and clearCache
will require that you have already built Alluxio (see about how to build Alluxio manually).
Some commands require the Alluxio cluster to be running, and others do not. Please check where each command specifies if it requires the Alluxio cluster to be running.