Using Dagster with Airflow#

The dagster-airflow package provides ways to integrate Airflow DAGs and Dagster pipelines.

Airflow Ingest#

This example demonstrates how to use make_dagster_pipeline_from_airflow_dag to compile an Airflow DAG into a Dagster pipeline that can be executed (and explored) the same way as a Dagster-native pipeline.

There are two pipelines in the repo:

• airflow_simple_dag demonstrates the use of Airflow templates.
• airflow_complex_dag shows the translation of a more complex dependency structure.

from airflow_ingest.airflow_complex_dag import complex_dag
from airflow_ingest.airflow_simple_dag import simple_dag
from dagster import repository
from dagster_airflow.dagster_pipeline_factory import make_dagster_pipeline_from_airflow_dag

airflow_simple_dag = make_dagster_pipeline_from_airflow_dag(simple_dag)
airflow_complex_dag = make_dagster_pipeline_from_airflow_dag(complex_dag)


@repository
def airflow_ingest_example():
    return [airflow_complex_dag, airflow_simple_dag]

Note that the "execution_date" for the Airflow DAG is specified through the pipeline tags. To specify tags, call to:

make_dagster_pipeline_from_airflow_dag(
    dag=dag,
    tags={'airflow_execution_date': utc_execution_date_str}
)