I am a Google Cloud PM, but not on Dataflow specifically.
Do you know how long the Dataflow processing will take? You may be able to estimate the length of the full-size process (assuming it's batch rather than streaming) by running on a small data sample to see how many vCPU hours it takes and estimate from there, assuming your data takes a uniform amount of time to process.
Since Dataflow is running code that you wrote, it doesn't know how complex your processing will be for any given value, so it's hard to give any reasonable estimate. In general, automatically predicting how long an arbitrary piece of code will take to run on a given input is even harder than the Halting problem (https://en.wikipedia.org/wiki/Halting_problem), which is already undecidable.
Another way to look at Dataflow pricing as it related to this type of technology. Let's just assume for a second Dataflow and other similar technologies execute your pipeline on your data within the same resource-time amount.
Dataflow lets you set an upper bound on these resources, and lets you auto-scale, with essentially per-minute granularity. Alternatively, you can create a cluster of fixed resource amount. Cost of Dataflow is $0.01 per hour per CPU, in addition to those resources.
Dataflow's model hence either guarantees 100% resource efficiency, or you specifically opt into the upper bound of spend. A deployment model that lets you "spin up a cluster and pay for it" is intrinsically less efficient.
I wrote a blog on this WRT BigQuery, but Dataflow is right there [0].
Thanks. I know it's tricky due to scaling, etc, but I guess in the same way dataflow works out when data should arrive by (this is a streaming task btw), an estimate of the daily cost could be calculated and would be useful.
