Create Event Driven Airflow Pipeline with SNS, SQS and Lambda

Not a Medium member? Click here to read for free.

In my previous article, Create Event Driven Airflow Pipeline with Amazon SQS, I tried to create a solution to trigger Airflow DAGs using the message content from an SQS queue. However, I did not like that the SqsSensor operator will get activated after the DAG has been triggered an I wanted to use the messages in the queue to trigger the DAG instead.

Architecture

Set Up

SNS

I created an SNS topic named transactions. I selected the Standard type and left every other setting to the default.

SQS

I created a queue called trigger_queue and subscribed it to the transactions SNS topic.

Subscribe queue to transactions SNS topic

To test that your queue can receive messages from the transactions topic, go Amazon SNS > Topics > transactions > Publish message and build a simple message payload.