BigQuery Sharded Tables: Beware Schema Changes Affecting Data Transfer and Queries
5 months ago
Sharded tables in BigQuery can be problematic when the schema changes, such as adding columns or altering data types. These changes can disrupt data loading and querying using wildcards, as BigQuery uses the latest metadata for all shards, making it difficult to union shards with different schemas. This issue was highlighted when date columns in BigQuery Data Transfer for Facebook changed. Partitioned and clustered tables are recommended alternatives.