Cloud Composer Google
An Apache Airflow-based workflow orchestration solution that is completely managed.
Composer Cloud
Advantages
Completely controlled orchestration of workflows
Because Composer Cloud is managed, and it is compatible with Apache Airflow, you can concentrate on composing, scheduling, and monitoring your workflows instead of worrying about resource provisioning.
Connects to more Google Cloud goods
Users may fully orchestrate their pipeline thanks to end-to-end interaction with Google Cloud products, such as BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub, and AI Platform.
Allows for multiple and hybrid clouds
Regardless of whether your pipeline is entirely hosted on Google Cloud, exists in various clouds, or is on-premises, you can author, plan, and monitor your workflows with just one orchestration tool.
Google Cloud Composer
Two-way and multicloud
Organise workflows that go back and forth between on-premises and public cloud to facilitate your cloud migration or to keep your hybrid data environment running well. To get a unified data environment, create workflows that link data, processing, and services across clouds.
Accessible
Because it is based on Apache Airflow, users can be mobile and not be locked into any one platform. Customers can avoid lock-in with this open source project that Google is donating back to, and it integrates with a wide range of platforms that will only increase in number as the Airflow community grows.
Simple orchestration
Python is used to easily configure Cloud Composer pipelines as directed acyclic graphs (DAGs), accessible to all users. Troubleshooting is made simple with one-click deployment, which provides rapid access to a comprehensive library of connectors and several graphical depictions of your workflow in operation. Your directed acyclic graphs automatically synchronise to keep your jobs on time.
Cloud composer documentation
About Cloud Composer
With Cloud Composer, a fully managed workflow orchestration solution, you can plan, organise, oversee, and control workflow pipelines that connect on-premises data centres and clouds.
Cloud Composer runs on the Python programming language and is based on the well-known Apache Airflow open source project.
You can take advantage of all the features of Apache Airflow without any installation or administrative work by switching to it from a local instance. You can focus on your workflows rather than your infrastructure by using Cloud Composer to easily create managed Airflow environments and use Airflow-native features like the robust Airflow web interface and command-line tools.
Version differences for Cloud Composer
Major Cloud Composer versions
The following are the major versions of Composer Cloud:
Cloud Composer 1: you can manually scale the environment and deploy the infrastructure to your networks and projects.
Cloud Composer 2: The environment’s cluster in this version automatically adjusts to the demands on its resources.
Cloud Composer 3: This version hides infrastructure elements, such as the environment’s cluster and dependencies on other services, and simplifies network configuration.
Workflows known as Airflow and Airflow DAG
A workflow in data analytics is a set of operations for obtaining, processing, evaluating, or using data. Workflows in Airflow are generated using the use of DAGs, or “Directed Acyclic Graphs”.
A directed acyclic graph (DAG) is a set of tasks that you wish to plan and execute, arranged to show their dependencies and relationships. Python files are used to generate DAGs, and these files use code to specify the DAG structure. Ensuring that each work is completed on time and in the correct order is the goal of the DAG.
In a DAG, every task can represent nearly anything. For instance, a single task could carry out any of the following tasks:
- Getting ready for data intake
- Keeping an eye on an API
- Transmitting an electronic message
- Managing a pipeline
You can manually start DAGs or have them run in reaction to events, such modifications in a Cloud Storage bucket, in addition to scheduling DAGs. Refer to Triggering DAGs for additional details.
Environments for Cloud Composer
Self-contained Airflow deployments built on Google Kubernetes Engine are known as Cloud Composer environments. They use Airflow’s built-in connectors to interface with other Google Cloud services. In a single Google Cloud project, you can establish one or more environments in any supported area.
The Google Cloud services that power your workflows and every Airflow component are provisioned via Cloud Composer. An environment’s primary constituents are:
- GKE cluster: Airflow schedulers, triggerers, workers, and other airflow components process and execute DAGs by running as GKE workloads in a single cluster that is customised for your environment.
- In addition, the cluster is home to additional Composer components including Composer Agent and Airflow Monitoring, which collect metrics to be uploaded to Cloud Monitoring, logs to be stored in Cloud Logging, and assist in managing the Composer environment.
- Apache Airflow UI is run by the web server known as the Airflow web server.
- Airflow database: The Apache Airflow metadata is stored in this database.
- Cloud Storage bucket: A Cloud Storage bucket is linked to your environment by Cloud Composer. The DAGs, logs, customised plugins, and environment data are all kept in this bucket, which is also referred to as the environment’s bucket.
Interfaces for Cloud Composer
Interfaces for controlling environments, individual DAGs, and Airflow instances running within environments are provided by this.
For instance, you can use Terraform, the Cloud Composer API, the Google Cloud CLI, or the Google Cloud console to construct and configure this environments.
As a further example, you can use the Google Cloud console, the native Airflow UI, or the Google Cloud CLI and Airflow CLI commands to manage DAGs.
Features of Cloud Composer’s airflow
You can control and make use of Airflow capabilities like these with Cloud Composer:
- Airflow DAGs: Using the native Airflow UI or the Google Cloud console, you may add, modify, remove, or trigger Airflow DAGs.
- Airflow configuration options: You can set custom settings in place of Cloud Composer’s default values for these configuration options. Certain configuration parameters in Cloud Composer are blocked, meaning that you are unable to modify their values.
- Airflow Intersections.
- Airflow User Interface.
- Airflow Command Line Interface.
- Custom plugins: In your Cloud Composer environment, you can install custom Airflow plugins, such as hooks, sensors, interfaces, or custom, in-house Apache Airflow operators.
- Python dependencies: You can install dependencies from private package repositories, such as those found in the Artefact Registry, or from the Python Package Index within your environment. Plugins are another option if the dependencies are not listed in the package index.
- Recording and observing DAGs, Airflow elements, and Cloud Composer settings:
- The logs folder in the environment’s bucket and the Airflow web interface both allow you to view Airflow logs linked to individual DAG tasks.
- Environment metrics and cloud monitoring logs for Cloud Composer environments.
Controlling access in Cloud Composer
At the Google Cloud project level, security is managed by you, and you can designate IAM roles that let specific users build or edit environments. Someone cannot access any of your environments if they do not have the necessary Cloud Composer IAM role or access to your project.
You can also use Airflow UI access control, which is based on the Apache Airflow Access Control concept, in addition to IAM.
Cloud Composer Costs
You only pay for the amount of usage with Cloud Composer, which is expressed in terms of vCPU/hour, GB/month, and GB transferred/month. Because it employs numerous Google Cloud products as building pieces, we have multiple pricing tiers.
All levels of consumption and continuous usage are priced equally. See the price page for further details.
