Google has rolled out new features in BigQuery that allow users to view, trigger, and pause Airflow DAGs (Directed Acyclic Graphs) directly from the BigQuery interface. This feature is currently in Preview.
Key Features:
DAG Overview: Users can view all Airflow DAGs from Cloud Composer 3 environments in their Google Cloud project.
Detailed DAG Information:
- View individual DAG details
- Access past DAG runs (filterable by time range)
- Visualize DAG task dependencies through a diagram
- Examine DAG source code
DAG Management:
- Manually trigger DAGs for one-time runs
- Pause DAG execution
User Interface Enhancements:
- Customizable column display for additional DAG details
- Refresh option to update displayed data
Integration: This feature bridges BigQuery and Cloud Composer, streamlining workflow management.
How to Access:
- Navigate to the Orchestration page in the Google Cloud console.
- Select a DAG to view its details, runs, diagram, or code.
This update aims to provide BigQuery users with more direct control over their data workflows, potentially improving efficiency in managing complex data pipelines.