BigQuery Enhances Anomaly Detection for Multivariate Time Series Models

August 21, 2024 at 11:25:09 AM

TL;DR BigQuery ML supports anomaly detection with multivariate time series (ARIMA_PLUS_XREG) models for historical and new data. The ML.DETECT_ANOMALIES function uses ARIMA_PLUS and ARIMA_PLUS_XREG for time series data and autoencoder, k-means, or PCA for IID data. The CREATE MODEL statement creates these models, handling data frequency, irregular intervals, missing data, outliers, and seasonal patterns using various techniques.

BigQuery Enhances Anomaly Detection for Multivariate Time Series Models

You can now use BigQuery ML's multivariate time series ARIMA_PLUS_XREG models for anomaly detection. This feature allows for detecting anomalies in both historical and new data with multiple feature columns and is generally available (GA). You can explore this feature through the "Perform anomaly detection with a multivariate time-series forecasting model" tutorial.

ML.DETECT_ANOMALIES Function

The ML.DETECT_ANOMALIES function in BigQuery ML enables anomaly detection for time series data using ARIMA_PLUS and ARIMA_PLUS_XREG models. For independent and identically distributed random variables (IID) data, it uses autoencoder, k-means, or principal component analysis (PCA) models.

CREATE MODEL Statement for ARIMA_PLUS_XREG Models

The CREATE MODEL statement is used to create multivariate time series models in BigQuery. Forecasting is performed when the model is created, and you can retrieve forecasting values and compute prediction intervals using the ML.FORECAST and ML.EXPLAIN_FORECAST functions.

Time Series Modeling Pipeline

The multivariate ARIMA_PLUS_XREG time series model includes linear external regressors. The modeling pipeline for ARIMA_PLUS time series models performs several functions:

bigquery Time series modeling pipeline

  • Infer data frequency
  • Handle irregular time intervals
  • Handle duplicated timestamps by averaging
  • Interpolate missing data using local linear interpolation
  • Detect and clean spike and dip outliers
  • Detect and adjust abrupt step changes
  • Detect and adjust holiday effects
  • Detect multiple seasonal patterns using Seasonal and Trend decomposition using Loess (STL)
  • Extrapolate seasonality using double exponential smoothing (ETS)
  • Detect and model trends using the ARIMA model and auto.ARIMA algorithm for automatic hyperparameter tuning, selecting the best model based on the lowest Akaike information criterion (AIC).

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources 👇

Related Posts

BigQuery Launches Vector Search and Vector Index Features

BigQuery Launches Vector Search and Vector Index Features

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Data Transfer Now Supports Incremental Teradata Migrations

BigQuery Data Transfer Now Supports Incremental Teradata Migrations

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Automate Your GA4 Audit - Say Goodbye to Manual Checks!

Automate Your GA4 Audit - Say Goodbye to Manual Checks!

Sponsored
GA4 Auditor
GA4 Auditor

Verified Sponsor

Verified Sponsor

GA4 Auditor is a Verified Sponsor. Want to get featured here? Contact us.

Verified Sponsor
BigQuery Now Supports GROUP BY and SELECT DISTINCT with Arrays and Structs

BigQuery Now Supports GROUP BY and SELECT DISTINCT with Arrays and Structs

BigQuery ML Integrates Anthropic Claude AI for Generative Text

BigQuery ML Integrates Anthropic Claude AI for Generative Text

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Introduces Python Code Completion with Gemini

BigQuery Introduces Python Code Completion with Gemini

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

Featured
GA4 Auditor logo

GA4 Auditor

Automated GA4 audits with actionable insights

Data Analysis
GA4 SQL logo

GA4 SQL

Generate GA4 BigQuery queries easily

Data Analysis
TapClicks logo

TapClicks

Automated marketing solutions powered by your data

Data Engineering
Stitch logo

Stitch

Automated cloud data pipelines, no coding needed

Data Engineering
Akkio logo

Akkio

AI-powered analytics for agencies

Data Analysis
Databricks logo

Databricks

Generative AI-powered data intelligence platform

Data Engineering
NinjaCat logo

NinjaCat

AI-powered marketing data and analytics platform

Reporting
Funnel logo

Funnel

Aggregate and analyze marketing data seamlessly

Reporting
Fivetran logo

Fivetran

Effortlessly centralize and move data from any source

Data Engineering