BigQuery Introduces Airflow DAG Management Features in Preview

October 08, 2024 at 5:47:05 AM

TL;DR Google has introduced new features in BigQuery to manage Airflow DAGs directly from the BigQuery interface, currently in Preview. Users can view, trigger, and pause DAGs from Cloud Composer 3 environments. Features include viewing DAG details, past runs, task dependencies, and source code. Users can manually trigger or pause DAGs, customize the interface, and refresh data. This integration streamlines workflow management.

BigQuery Introduces Airflow DAG Management Features in Preview

Google has rolled out new features in BigQuery that allow users to view, trigger, and pause Airflow DAGs (Directed Acyclic Graphs) directly from the BigQuery interface. This feature is currently in Preview.

Key Features:

  1. DAG Overview: Users can view all Airflow DAGs from Cloud Composer 3 environments in their Google Cloud project.

  2. Detailed DAG Information:

    • View individual DAG details
    • Access past DAG runs (filterable by time range)
    • Visualize DAG task dependencies through a diagram
    • Examine DAG source code
  3. DAG Management:

    • Manually trigger DAGs for one-time runs
    • Pause DAG execution
  4. User Interface Enhancements:

    • Customizable column display for additional DAG details
    • Refresh option to update displayed data
  5. Integration: This feature bridges BigQuery and Cloud Composer, streamlining workflow management.

How to Access:

  1. Navigate to the Orchestration page in the Google Cloud console.
  2. Select a DAG to view its details, runs, diagram, or code.

This update aims to provide BigQuery users with more direct control over their data workflows, potentially improving efficiency in managing complex data pipelines.

Have more questions on this topic? Ask our AI assistant for in-depth insights.

Read more from sources 👇

The Only Digital Marketing Feed You'll Ever Need.

Stay informed your way. Tailored updates when and how you want them. 100% Free.

10,000+ Users

500+ Sources

1000+ Tools

Or

Related Posts

Google Sheets Integrates BigQuery Saved Queries

Google Sheets Integrates BigQuery Saved Queries

Google
Google

Official Source

Official Source

Google is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Expands Search Index Capabilities with INT64 and TIMESTAMP Support

BigQuery Expands Search Index Capabilities with INT64 and TIMESTAMP Support

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Tired of spending too much time creating audits for your clients?

Tired of spending too much time creating audits for your clients?

Sponsored
Google Ends Free Access to Gemini in BigQuery, Announces Paid Plans

Google Ends Free Access to Gemini in BigQuery, Announces Paid Plans

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
Google Cloud Enhances Looker with Major User Experience Updates

Google Cloud Enhances Looker with Major User Experience Updates

Sean Zinsmeister
Sean Zinsmeister

Official Source

Official Source

Sean Zinsmeister is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Launches Gemini-Enhanced SQL Translation Features

BigQuery Launches Gemini-Enhanced SQL Translation Features

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Data Transfer Service Launches Data Source Change Log for Schema Updates

BigQuery Data Transfer Service Launches Data Source Change Log for Schema Updates

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source
BigQuery Introduces AI-Augmented Data Preparation with Gemini

BigQuery Introduces AI-Augmented Data Preparation with Gemini

Google Cloud
Google Cloud

Official Source

Official Source

Google Cloud is a Official Source. The source has been verified by Swipe Insight team.

Official Source

Related Tools

Marketing Auditor logo

Marketing Auditor

Verified Tool

Verified Tool

Marketing Auditor is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Automated audits for Google Ads and Analytics.

Get Featured Here

Showcase your tool in this list.

Contact Us
GA4 SQL logo

GA4 SQL

Verified Tool

Verified Tool

GA4 SQL is a Verified Tool. Want to get this badge? Contact us.

Verified Tool

Generate GA4 BigQuery queries easily

Data Analysis
TapClicks logo

TapClicks

Automated marketing solutions powered by your data

Data Engineering
Stitch logo

Stitch

Automated cloud data pipelines, no coding needed

Data Engineering
Akkio logo

Akkio

AI-powered analytics for agencies

Data Analysis
Databricks logo

Databricks

Generative AI-powered data intelligence platform

Data Engineering
NinjaCat logo

NinjaCat

AI-powered marketing data and analytics platform

Reporting
Funnel logo

Funnel

Aggregate and analyze marketing data seamlessly

Reporting
Fivetran logo

Fivetran

Effortlessly centralize and move data from any source

Data Engineering
Power My Analytics logo

Power My Analytics

Automate and integrate your marketing data

Reporting

Get Featured Here

Showcase your tool in this list.

Contact Us