Buying snapshot
Google Cloud Dataflow is a fully managed Google Cloud service for batch and streaming data processing, built on the open source Apache Beam SDK.
Scroll sideways to see all columns. With a keyboard, focus this table and use the arrow keys.
| Decision point | What Google states |
|---|---|
Pricing model | Google Cloud's pay-as-you-go pricing: no up-front fees and no termination charges; pricing varies by product and usage |
Deployment | Fully managed; runs on serverless Google Cloud infrastructure |
License | Google-provided Dataflow templates are published on GitHub under the Apache-2.0 license |
New-customer credit | New customers get $300 in free credits to spend on Dataflow |
Startup credit | Startups can get up to $350,000 in Cloud credits through the Google for Startups Cloud Program |
Custom quote | Google's sales team gives a custom quote for your organization |
On demand or committed use
Google offers Dataflow committed use discounts (CUDs) at two levels, depending on the commitment period. Each discount is set against the on-demand rate.
Scroll sideways to see all columns. With a keyboard, focus this table and use the arrow keys.
| Option | What Google states |
|---|---|
One-year CUD | 20% discount from the on-demand rate |
Three-year CUD | 40% discount from the on-demand rate |
How the bill is built
Google bills Dataflow compute resources in four parts. Which of them appear on a bill depends on the job: Shuffle data is billed for batch workloads, and the two Streaming Engine items cover Streaming Engine use. Rates depend on region; Google's pricing calculator estimates monthly Dataflow costs, including region specific pricing and fees.
Scroll sideways to see all columns. With a keyboard, focus this table and use the arrow keys.
| Billed item | Scope Google states |
|---|---|
Worker CPU and memory | Dataflow compute resources |
Dataflow Shuffle data processed | Batch workloads |
Streaming Engine Compute Units | Dataflow compute resources |
Streaming Engine data processed | Dataflow compute resources |
What drives the bill
Worker count. Dataflow autoscales in both batch and streaming pipelines and scales to 4K workers per job, so worker CPU and memory grow with the workers a job uses.
Costs outside the compute line
Disk, GPUs and snapshots. Google lists Persistent Disk, GPUs and snapshots as other Dataflow resources that are billed for all jobs.
Cost controls. Google Cloud offers budgets, alerts, quota limits and other free cost management tools, plus free 24/7 billing support. Dataflow also has a job cost monitoring UI for cost estimation.
Our take
If your Beam pipelines will run steadily for a year or more, Dataflow's published committed use discounts are the option to weigh; if volume is still changing, stay on demand until your usage settles.