# Trino on K8s [Trino](https://trino.io/) is an open source distributed SQL query engine for running interactive analytic queries on data at a large scale. This guide describes how to run queries against Trino with Alluxio as a distributed caching layer, for any data storage systems that Alluxio supports, such as AWS S3 and HDFS. Alluxio allows Trino to access data regardless of the data source and transparently cache frequently accessed data (e.g., tables commonly used) into Alluxio distributed storage. ## Prerequisites This guide assumes that the [Alluxio cluster is deployed on Kubernetes](https://documentation.alluxio.io/ee-da-en/da-3.5/start/install-alluxio-on-kubernetes). `docker` is also required to build the custom Trino image. ## Prepare Image To integrate with Trino, Alluxio jars and configuration files must be added within the Trino image. Trino containers need to be launched with this modified image in order to connect to the Alluxio cluster. Among the files listed in the [Alluxio installation instructions](https://documentation.alluxio.io/ee-da-en/da-3.5/start/install-alluxio-on-kubernetes#Preparation) to download, locate the tarball named `alluxio-enterprise-DA-3.5-10.2.0-release.tar.gz`. Extract the following Alluxio jars from the tarball: * `client/alluxio-DA-3.5-10.2.0-client.jar` * `client/ufs/alluxio-underfs-s3a-shaded-DA-3.5-10.2.0.jar` if using a S3 bucket as an UFS * `client/ufs/alluxio-underfs-hadoop-3.3-shaded-DA-3.5-10.2.0.jar` if using HDFS as an UFS Prepare an empty directory as the working directory to build an image from. Within this directory, create the directory `files/alluxio/` and copy the aforementioned jar files into it. Download the commons-lang3 jar into `files/` so that it can also be included in the image. ```console wget https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.14.0/commons-lang3-3.14.0.jar -O files/commons-lang3-3.14.0.jar ``` Create a `Dockerfile` with the operations to modify the base Trino image. The following example defines arguments for: * `TRINO_VERSION=449` as the Trino version * `UFS_JAR=files/alluxio/alluxio-underfs-s3a-shaded-DA-3.5-10.2.0.jar` as the path to the UFS jar copied into `files/alluxio/` * `CLIENT_JAR=files/alluxio/alluxio-DA-3.5-10.2.0-client.jar` as the path to the Alluxio client jar copied into `files/alluxio/` * `COMMONS_LANG_JAR=files/commons-lang3-3.14.0.jar` as the path to the commons-lang3 jar downloaded to `files/` ```shell ARG TRINO_VERSION=449 ARG IMAGE=trinodb/trino:${TRINO_VERSION} FROM $IMAGE ARG UFS_JAR=files/alluxio/alluxio-underfs-s3a-shaded-DA-3.5-10.2.0.jar ARG CLIENT_JAR=files/alluxio/alluxio-DA-3.5-10.2.0-client.jar ARG COMMONS_LANG_JAR=files/commons-lang3-3.14.0.jar # remove existing alluxio jars RUN rm /usr/lib/trino/plugin/hive/alluxio*.jar \ && rm /usr/lib/trino/plugin/delta-lake/alluxio*.jar \ && rm /usr/lib/trino/plugin/iceberg/alluxio*.jar # copy jars into image ENV ALLUXIO_HOME=/usr/lib/trino/alluxio RUN mkdir -p $ALLUXIO_HOME/lib && mkdir -p $ALLUXIO_HOME/conf COPY --chown=trino:trino $UFS_JAR /usr/lib/trino/alluxio/lib/ COPY --chown=trino:trino $CLIENT_JAR /usr/lib/trino/alluxio/lib/ COPY --chown=trino:trino $COMMONS_LANG_JAR /usr/lib/trino/alluxio/lib/ # symlink UFS jar to lib/ and client+commons-lang3 jar to each plugin directory RUN ln -s $ALLUXIO_HOME/lib/alluxio-underfs-*.jar /usr/lib/trino/lib/ RUN ln -s $ALLUXIO_HOME/lib/alluxio-*-client.jar /usr/lib/trino/plugin/hive/hdfs/ \ && ln -s $ALLUXIO_HOME/lib/alluxio-*-client.jar /usr/lib/trino/plugin/delta-lake/hdfs/ \ && ln -s $ALLUXIO_HOME/lib/alluxio-*-client.jar /usr/lib/trino/plugin/iceberg/hdfs/ RUN ln -s $ALLUXIO_HOME/lib/commons-lang3-3.14.0.jar /usr/lib/trino/plugin/hive/hdfs/ \ && ln -s $ALLUXIO_HOME/lib/commons-lang3-3.14.0.jar /usr/lib/trino/plugin/delta-lake/hdfs/ \ && ln -s $ALLUXIO_HOME/lib/commons-lang3-3.14.0.jar /usr/lib/trino/plugin/iceberg/hdfs/ USER trino:trino ``` This will copy the necessary jars into the image. For the UFS jar, a symlink is created in the Trino `lib/` directory pointing to it. For the Alluxio client and commons-lang3 jars, iterate through each of the possible plugin directories and create a symlink pointing to each of them. > In the above example, the jars are copied into the plugin directories for Hive, Delta Lake, and Iceberg, but only a subset may be needed depending on which connector(s) are being utilized. > Note that for Trino versions earlier than 434, the destination to copy the `CLIENT_JAR` and `COMMONS_LANG_JAR` should be `/usr/lib/trino/plugin//` instead of `/usr/lib/trino/plugin//hdfs/` Build the image by running, replacing `` with the URL of your private container registry and `` with the corresponding Trino version. ```console $ docker build --platform linux/amd64 -t /alluxio/trino: . ``` Push the image by running: ```console $ docker push /alluxio/trino: ``` ## Deploy Hive Metastore To complete the Trino end-to-end example, a Hive Metastore will be launched. If there is already one available, this step can be skipped. The Hive Metastore URI will be needed to configure the Trino catalogs. Use `helm` to create a stand-alone Hive metastore. Add the helm repository by running: ```console $ helm repo add heva-helm-charts https://hevaweb.github.io/heva-helm-charts/ ``` Install the Hive cluster by running: ```console $ helm install hive-metastore heva-helm-charts/hive-metastore ``` Check the status of the Hive pods: ```console $ kubectl get pods ``` It is known that the `hive-metastore-db-init-schema-*` pods result in an `Error` status, but this does not impact the rest of the workflow. When completed, delete the Hive cluster by running: ```console $ helm uninstall hive-metastore ``` ### Configure AWS credentials for S3 If using S3, provide AWS credentials by editing the configmap: ```console $ kubectl edit configmap hive-metastore ``` In the editor, search for the `hive-site.xml` properties `fs.s3a.access.key` and `fs.s3a.secret.key` and populate them with AWS credentials to access to your S3 bucket. Restart the Hive Metastore pod by deleting it: ```console $ kubectl delete pod hive-metastore-0 ``` And it will restart automatically. ## Configure Trino Create a `trino-alluxio.yaml` configuration file in the installation directory to define the Alluxio specific configurations. We'll break down this configuration file into several parts that can be joined into a single yaml file. ### Setting image and server ```yaml image: registry: repository: tag: server: workers: 2 log: trino: level: INFO config: query: maxMemory: "4GB" ``` Specify the location of the custom Trino image that was previously built and pushed. Customize the Trino server specifications as needed. ### Setting additionalCatalogs ```yaml additionalCatalogs: memory: | connector.name=memory memory.max-data-per-node=128MB ``` In addition to memory related configurations, the catalog configurations are added under this section. Depending on which connector will be used, add the corresponding catalog configuration under `additionalCatalogs`. If following the previous section to launch the Hive Metastore, the value of `hive.metastore.uri` will be `thrift://hive-metastore.default:9083`.
Hive ```yaml # connectors using alluxio hive: | connector.name=hive fs.hadoop.enabled=true # use the uri of your hive-metastore deployment hive.metastore.uri=thrift:// hive.config.resources=/etc/trino/alluxio-core-site.xml hive.non-managed-table-writes-enabled=true hive.s3-file-system-type=HADOOP_DEFAULT ``` > If using Trino version 458 or later, `fs.hadoop.enabled=true` is required
Delta Lake ```yaml # connectors using alluxio delta_lake: | connector.name=delta_lake fs.hadoop.enabled=true hive.metastore.uri=thrift:// hive.config.resources=/etc/trino/alluxio-core-site.xml hive.s3-file-system-type=HADOOP_DEFAULT delta.enable-non-concurrent-writes=true delta.vacuum.min-retention=1h delta.register-table-procedure.enabled=true delta.extended-statistics.collect-on-write=false ``` > If using Trino version 458 or later, `fs.hadoop.enabled=true` is required
Iceberg ```yaml # connectors using alluxio iceberg: | connector.name=iceberg fs.hadoop.enabled=true hive.metastore.uri=thrift:// hive.config.resources=/etc/trino/alluxio-core-site.xml ``` > If using Trino version 458 or later, `fs.hadoop.enabled=true` is required
### Setting coordinator and worker The `coordinator` and `worker` configurations both require similar modifications to their `additionalJVMConfig` and `additionalConfigFiles` sections. For the coordinator: ```yaml coordinator: jvm: maxHeapSize: "8G" gcMethod: type: "UseG1GC" g1: heapRegionSize: "32M" config: memory: heapHeadroomPerNode: "" query: maxMemoryPerNode: "1GB" additionalJVMConfig: - "-Dalluxio.home=/usr/lib/trino/alluxio" - "-Dalluxio.conf.dir=/etc/trino/" - "-XX:+UnlockDiagnosticVMOptions" - "-XX:G1NumCollectionsKeepPinned=10000000" - "-Djava.security.manager=allow" additionalConfigFiles: alluxio-core-site.xml: | fs.s3a.impl alluxio.hadoop.FileSystem fs.hdfs.impl alluxio.hadoop.FileSystem alluxio-site.properties: | alluxio.etcd.endpoints: http://:2379 alluxio.cluster.name: default-alluxio alluxio.k8s.env.deployment: true alluxio.mount.table.source: ETCD alluxio.worker.membership.manager.type: ETCD metrics.properties: | # Enable the Alluxio Jmx sink sink.jmx.class=alluxio.metrics.sink.JmxSink ``` > If using Trino version 464 or later, `"-Djava.security.manager=allow"` is required to be compatible with Java 23 Note the 2 additional JVM configurations to define the `alluxio.home` and `alluxio.conf.dir` properties. Under `additionalConfigFiles`, 3 Alluxio configuration files are defined. * `alluxio-core-site.xml`: The xml file defines which underlying Filesystem class should be used for specific schemes of file URIs. By setting the value `alluxio.hadoop.FileSystem`, this ensures the file is executed through the Alluxio filesystem. Depending on the UFS, set the corresponding configurations: * For S3, set `fs.s3a.impl` to `alluxio.hadoop.FileSystem` * For HDFS, set `fs.hdfs.impl` to `alluxio.hadoop.FileSystem` * `alluxio-site.properties`: The properties file defines the Alluxio specific configuration properties for the Alluxio client. The next section will describe how to populate this file. * `metrics.properties`: Adding this one line enables Alluxio specific metrics to be scraped from Trino. For the worker, include the same 2 additional JVM configurations and copy the same 3 Alluxio configuration files under `additionalConfigFiles`. ```yaml worker: jvm: maxHeapSize: "8G" gcMethod: type: "UseG1GC" g1: heapRegionSize: "32M" config: memory: heapHeadroomPerNode: "" query: maxMemoryPerNode: "1GB" additionalJVMConfig: - "-Dalluxio.home=/usr/lib/trino/alluxio" - "-Dalluxio.conf.dir=/etc/trino/" - "-XX:+UnlockDiagnosticVMOptions" - "-XX:G1NumCollectionsKeepPinned=10000000" - "-Djava.security.manager=allow" additionalConfigFiles: alluxio-core-site.xml: | fs.s3a.impl alluxio.hadoop.FileSystem fs.hdfs.impl alluxio.hadoop.FileSystem alluxio-site.properties: | alluxio.etcd.endpoints: http://:2379 alluxio.cluster.name: default-alluxio alluxio.k8s.env.deployment: true alluxio.mount.table.source: ETCD alluxio.worker.membership.manager.type: ETCD metrics.properties: | sink.jmx.class=alluxio.metrics.sink.JmxSink ``` > If using Trino version 464 or later, `"-Djava.security.manager=allow"` is required to be compatible with Java 23 #### alluxio-site.properties For the Trino cluster to properly communicate with the Alluxio cluster, certain properties must be aligned between the Alluxio client and Alluxio server. To show `alluxio-site.properties` from the Alluxio cluster config map, run ```console $ kubectl get configmap alluxio-alluxio-conf -o yaml ``` where `alluxio-alluxio-conf` is the name of the configmap for the Alluxio cluster. In this output, search for the following properties under `alluxio-site.properties` and set them for the coordinator's and worker's `alluxio-site.properties`: * `alluxio.etcd.endpoints`: The ETCD hosts of the Alluxio cluster. Assuming the hostnames are accessible from the Trino cluster, the same value can be used for the Trino cluster's `alluxio-site.properties`. * `alluxio.cluster.name`: This must be set with the exact same value for Trino cluster. * `alluxio.k8s.env.deployment=true` * `alluxio.mount.table.source=ETCD` * `alluxio.worker.membership.manager.type=ETCD` If following the [Install Alluxio on Kubernetes](https://documentation.alluxio.io/ee-da-en/da-3.5/start/install-alluxio-on-kubernetes) instructions, the values should be: ```yaml alluxio-site.properties: | alluxio.etcd.endpoints: http://alluxio-etcd.default:2379 alluxio.cluster.name: default-alluxio alluxio.k8s.env.deployment: true alluxio.mount.table.source: ETCD alluxio.worker.membership.manager.type: ETCD ``` ### Launch Trino Add the helm chart to the local repository by running: ```console $ helm repo add trino https://trinodb.github.io/charts ``` To deploy Trino, run: ```console $ helm install -f trino-alluxio.yaml trino-cluster trino/trino ``` Check the status of the Trino pods and identify the pod name of the Trino coordinator: ```console $ kubectl get pods ``` When completed, delete the Trino cluster by running: ```console $ helm uninstall trino-cluster ``` ## Executing Queries To enter the Trino coordinator pod, run: ```console $ kubectl exec -it $POD_NAME -- /bin/bash ``` where `$POD_NAME` is the full pod name of the Trino coordinator, starting with `trino-cluster-trino-coordinator-` Inside the Trino pod, run: ```console $ trino ``` To create a simple schema in your UFS mount, run ```console trino> CREATE SCHEMA hive.example WITH (location = ''); ``` where `UFS_URI` is a path within one of the defined Alluxio mounts. Using a S3 bucket as an example, this could be `s3a://myBucket/myPrefix/`.