Dataproc autoscaling for Trino workloads
An open-source, widely used distributed SQL query engine for warehouses and data lakes is called Trino. Numerous companies use it to examine big datasets kept in cloud storage, other data sources, including the Hadoop Distributed File System (HDFS).
Cluster setup and management are made simple with Dataproc, a managed Hadoop and Spark service. However, workloads like Trino that aren’t built on Yet Another Resource Negotiator, or YARN, aren’t yet supported by Dataproc for autoscaling.
Self-scaling By addressing the absence of autoscaling support for workloads that are not YARN-based, Dataproc for Trino helps to avoid overprovisioning, underprovisioning, and manual scaling. By autonomously scaling clusters in response to workload needs, it lowers operational strain, enhances query performance, and saves cloud expenses.
As a result, Dataproc becomes a more alluring platform for Trino workloads, allowing for real-time fraud detection, risk assessment, and analytics. They provide a technique in this blog post that allows Trino to automatically scale while it is operating on a Dataproc cluster.
Hadoop and Trino
Big data sets may be processed and saved in a manner that distributes over a network of personal computers using the free Hadoop software framework. It offers a distributed computing platform for large data processing that is dependable, scalable, and adaptable. A YARN centralized resource manager is used by Hadoop for resource allocation, cluster management, and monitoring.
Trino allows users to query data in diverse formats and from different sources using a single SQL interface by using a variety of data sources, including Hadoop, Hive, and other data lakes and warehouses.
The Trino Coordinator, who oversees planning, resource allocation, and query coordination, is in charge of Trino’s resource allocation and administration. For every query, Trino dynamically allots fine-grained CPU and memory resources. Trino clusters often depend on third-party cluster management platforms, such as Kubernetes, for scalability and resource distribution. These systems manage the dynamic scaling and provisioning of cluster resources. On Hadoop clusters, Trino does not utilize YARN for resource allocation.
Dataproc and Trino Dataproc is a managed Hadoop and Spark service that offers large data workloads on Google Cloud a completely managed environment. As of right now, Dataproc can only handle autoscaling for YARN-based apps. since of this, it is difficult to optimize the expenses of operating Trino on Dataproc since the cluster size has to be changed to accommodate for the processing demands of the moment.
Without sacrificing workload execution, the Autoscaler for Trino on Dataproc solution offers dependable autoscaling for Trino on Dataproc.
Trino presents obstacles
Trino’s embedded discovery service is used in the Trino deployment on Dataproc. At initialization, every Trino node establishes a connection with the discovery service and sends out periodic heartbeat signals.
The worker registers with the discovery service upon joining the cluster, enabling the Trino coordinator to begin assigning new tasks to the newly added workers. However, in the event that a worker abruptly stops functioning, it may be challenging to remove them from the cluster, perhaps leading to total query failure.
Trino offers a graceful shutdown API that should only be used on workers to guarantee that they end without interfering with ongoing requests. The worker is placed in a SHUTTING_DOWN state via the shutdown API, and the coordinator ceases to assign new tasks to the workers. The worker will continue to do any tasks that are pending in this condition, but it won’t take on any new ones. The Trino worker will leave after every running job has completed.
Because of this Trino worker behavior, workers must be watched over by the Trino Autoscaler solution to make sure they gracefully quit before the VMs are removed from the cluster.
Method of solving the problem
The solution tracks the CPU utilization of the cluster and the specifics of the secondary worker nodes with the least amount of CPU use by querying the Cloud Monitoring API. There is a cooldown time in between each scaling action, during which no further scaling actions are performed. Based on worker node count and CPU consumption, the cluster is scaled up or down.
Taking into Account
- Decisions on cluster size are based on total CPU usage, and the secondary worker node with the lowest CPU utilization determines which node should be eliminated.
- By default, secondary worker nodes are preemptive virtual machines (VMs). Changing the size of the cluster only affects these VMs, not the HDFS workloads.
- The coordinator node is where the program runs, and Dataproc has autoscaling turned off by default.
- The hiring of additional personnel will only benefit newly submitted jobs; current positions will continue to be filled by bound individuals.
In Summary
Your Dataproc cluster may be automatically scaled depending on workload, ensuring that you only utilize the resources you need. Significant cost reductions are possible with autoscaling, particularly for workloads with erratic demand.
