should be similar performance
But only with Iceberg? Or also with other formats
obviously it’ll depend on the workload, but learning more about Iceberg made me understand BigQuery better. They’re clearly based on similar formats and styles
I see, I will give it a look. Already moving to physical storage we saved 10x on a few massive tables
Iceberg has the best BQ support
https://cloud.google.com/bigquery/docs/query-iceberg-data
If we 10x the data we definitely will look for more optimization
BQ has a private preview running for Managed Iceberg, with several plans moving forward
https://www.youtube.com/watch?v=4d4nqKkANdM&ab_channel=ApacheIceberg
the other reason for us is the ability to use other query engines. our biggest cost is query analysis, so being able to move some queries to DuckDB, StarRocks, or another engine, could save us a boatload
Super interesting - latency for adhoc analysis is always our concern - e.g. an Analysts time waiting for query response & data engineer cost to spin up the federated data source vs just doing it through BQ
Regarding the BQ colab notebook, does anyone have any good practice material?
