ExternalTaskSensor vs TriggerDagRunOperator

Not a Medium member? Click here to read for free.

I have been exploring airflow sensors and deferrable operators in my recent articles. In the last article, I wrote about how to Create Event Driven Airflow Pipeline with SNS, SQS and Lambda which was a sequel to Create Event Driven Airflow Pipeline with Amazon SQS.

In this article, I want to answer more queries that look like these:

• How to trigger Airflow DAG on the success of another DAG
• How to pause Airflow DAG until another DAG is completed
• How to trigger Airflow DAG from another DAG with config
• How to use TriggerDagRunOperator with dynamic tasks

This time around, I will use the TriggerDagRunOperator and ExternalTaskSensor classes.

TriggerDagRunOperator

The simplest way to use this is to provide it with the DAG ID for the target DAG.

# ets_vs_tdr_trigger
trigger_task = TriggerDagRunOperator(
        task_id='trigger_task_1',
        trigger_dag_id='ets_vs_tdr_receiver'
    )

I am starting by setting up two DAGs: ets_vs_tdr_trigger will be used to trigger ets_vs_tdr_receiver.

Using Custom Run ID

When I use triggers like this, I like to customize the run ID in the target DAG so that when looking through DAG runs later on I know whether this DAG was a…