BigQuery Introduces ML.DESCRIBE Function for Enhanced Data Analysis

The ML.DESCRIBE() function in BigQuery provides a statistical overview of a table or query, similar to the df.describe() function in pandas. It offers detailed analysis elements including the minimum, maximum, mean, median, number of unique values, quantiles, top elements, and more. It also handles STRUCT and ARRAY data types by expanding STRUCTs and unnesting ARRAYs before computation.

While ML.DESCRIBE() in BigQuery is similar to pandas' df.describe() in providing a statistical overview, it goes further by offering a more extensive set of analysis elements. Additionally, BigQuery's ML.DESCRIBE() function takes into account STRUCT and ARRAY data types by expanding and unnesting them, which is not a functionality provided by pandas.

Alongside ML.DESCRIBE(), BigQuery released several other functions, including:

• ML.VALIDATE_DATA_SKEW: Computes statistics to compare serving data to training data and identify anomalous differences.
• ML.VALIDATE_DATA_DRIFT: Computes statistics for two sets of serving data to identify anomalous differences.
• ML.TFDV_DESCRIBE: Computes fine-grained descriptive statistics for training or serving data (similar to tfdv.generate_statistics_from_csv API).
• ML.TFDV_VALIDATE: Computes statistics for training and serving data or two sets of serving data to identify anomalous differences (similar to tfdv.validate_statistics API).