Google Dataflow Streaming
Real-time inventory management, fraud prevention, and patient monitoring are just a few of the analytics and machine learning use cases that Dataflow’s industry-leading data processing platform can handle. It also offers unified batch and streaming capabilities. It is a fully managed service that offers data engineers a wide range of integrated management tools, giving them flexibility and choice. Flexible development options include pre-built templates, notebooks, and SDKs for Go, Python, and Java. BigQuery, Vertex AI, Cloud Storage, Spanner, BigTable, Pub/Sub, and other Google Cloud products are all integrated with Dataflow. To best meet your analytics and machine learning needs, it also integrates with third-party services like AWS S3 and Snowflake, as well as open-source technologies like Kafka and JDBC.
Customers with predictable processing volumes want to better optimize their Dataflow costs as the needs for machine learning and streaming analytics continue to grow. They are pleased to announce that Dataflow streaming committed use discounts (CUDs) are now generally available. This offers you a new opportunity to reduce the cost of streaming compute, which is a major contributor to your streaming expenses. You can receive greater savings by committing to a minimum level of Dataflow streaming compute usage for one or three years: a one-year commitment will result in a 20% discount, and a three-year commitment will result in a 40% discount.
Spend-based commitments known as dataflow streaming CUDs are applicable to the following dataflow resources for all projects or regions linked to a single Cloud Billing account:
- Worker memory and CPU for streaming tasks
- Use of streaming engines
- Dataflow Prime streaming jobs using data compute units (DCUs)
You can buy dataflow streaming CUDs through the Google Cloud console.
When to employ CUDs for Dataflow streaming
Workloads with predictable resource requirements, such as smart supply chains, predictive maintenance, and personalized product recommendations, are best suited for dataflow streaming CUDs. You can buy Dataflow streaming CUDs to better manage the cost of your streaming jobs, as they are expected to be used fairly frequently.
Since adding more workloads to Dataflow streaming increases the value that existing workloads can get from Dataflow streaming CUDs, Dataflow streaming CUDs are also a good option for workloads that are growing steadily. For instance, if you anticipate a 30% annual growth in your streaming usage, you can lock in a 40% discount by purchasing a three-year Dataflow streaming CUD at your current usage levels. You can add more Dataflow streaming CUDs as your workload increases to increase or maintain your CUD utilization rate and receive even more discounts on your Dataflow streaming expenditure.
CUDs for Dataflow streaming
Visit the Cloud console’s dedicated use discounts page to buy a Dataflow streaming CUD. Based on the anticipated growth and usage trends of the streaming jobs you would like to cover with your commitment, you can determine how much you are willing to commit. You will see the cost of your commitment after the discounts and the amount you are saving as soon as you select your commitment period. It really is that simple!
Dataflow Streaming costs
For Dataflow streaming jobs, committed use discounts (CUDs) offer reduced pricing in return for your promise to consistently use a predetermined volume of Dataflow compute resources for a minimum of one year.
When you can commit to a consistent minimum spending on Dataflow compute capacity for streaming jobs for a minimum of one year, then Dataflow CUDs are the best option.
Dataflow CUD costing
Two discount levels are available from Dataflow CUDs based on the length of the commitment:
- You can save 20% on the on-demand rate with a one-year CUD.
- A CUD of three years saves you forty percent over the on-demand price.
A commitment can be purchased for one or three years, and the commitment amount you select determines how much you expect Dataflow to spend on compute capacity per hour during that time. Once your discount has been applied, this commitment amount becomes your commitment fee. For the duration of the commitment period, you will then receive monthly bills for this commitment fee. See the section of this document titled “An example Dataflow CUD” for a complete example.
Regardless of instance configuration or region, the discount is valid for any eligible usage in Dataflow projects linked to the Cloud Billing account used to purchase the commitment. For both regional and multi-region configurations, all CUDs are applicable.
Any additional costs are charged at the on-demand rate if they exceed the commitment. You can buy more commitments as your Dataflow usage increases to get discounts on rising costs that aren’t covered by earlier commitments.
Your commitment fee is unaffected if Dataflow’s on-demand rates change after you purchase a commitment. On applicable usage, you get the same percentage off.
Resources that qualify for CUDs in Dataflow
Your expenditures on the Dataflow compute capacity utilized by streaming jobs across projects will automatically be reduced by Dataflow committed use discounts. With this flexibility, you can save time and money by achieving a high utilization rate of your commitment across projects and regions without the need for manual intervention. When you spend money on the following resources, Dataflow CUDs are applicable:
- Worker memory and CPU for streaming tasks
- Processed data from streaming engines
- Compute units for streaming engines
- DCUs (data compute units) for streaming applications
The following resources are not covered by Dataflow CUDs:
- CPU and memory usage by workers for batch and FlexRS tasks
- Dataflow shuffle the processed data
- Batch job data compute units (DCUs)
- Permanent Disk Retention
- GPUs
- Moments in time
- Secret Virtual Machines
Proceed with the following action
A flexible, scalable, and affordable platform for managing your streaming workloads is offered by Dataflow. You can reduce your Dataflow expenses even further by using Dataflow streaming CUDs! For further information on Dataflow streaming CUDs, visit our pricing page and documentation pages.
