BigQuery introduces several key updates enhancing data transfer, regional flexibility, and vector indexing capabilities.
Automated Data Transfers from Snowflake to BigQuery (Preview)
BigQuery Data Transfer Service now supports scheduling automated data transfers from Snowflake. This Snowflake connector allows users to schedule and manage migration jobs using public IP allow lists. The process involves migration agents running on Google Kubernetes Engine, which trigger data loads from Snowflake to a staging area within the same cloud provider. For Snowflake accounts hosted on AWS, data is first staged in an Amazon S3 bucket before being transferred to BigQuery. This feature is currently in preview.
Cross-Region Transfers for Batch Loading and Exporting (GA)
BigQuery now supports cross-region transfers for batch loading and exporting data. Users can load or export data between any regions or multi-regions using commands like bq load, LOAD DATA, bq extract, or EXPORT DATA. This functionality is generally available, providing greater flexibility in managing data across geographic locations.
Vector Indexes with TreeAH Index Type (GA)
BigQuery's vector indexes now support the TreeAH index type, which leverages Google's ScaNN algorithm. This index type is optimized for efficient batch processing, capable of handling from a few thousand up to hundreds of thousands of embeddings simultaneously. This enhancement is generally available, improving performance for large-scale vector similarity searches.
