Trigger Airflow DAG with REST API using Snowflake

In an earlier article I wrote about How to Trigger Airflow DAG Using REST API. I have also extended that concept with the article where I highlighted how to Create An Event Driven Airflow Pipeline with SNS, SQS and Lambda.

Not a Medium member? Click here to read for free!

In this demo, I will create a connection between Snowflake and Airflow such that we can have a Snowflake object trigger an Airflow DAG.

Solution architecture

The steps are as follows:

1. Create a Lambda function
2. Create an API gateway in AWS
3. Create the required permissions in AWS
4. Create an API integration in Snowflake. Then update AWS role from (3) above with Snowflake details.
5. Create the Snowflake external function to call Lambda

Take note of the following:

• I am reusing Airflow REST API setup instructions from my article How to Trigger Airflow DAG Using REST
• I am reusing the DAG sns_receiver_dag from my article Create An Event Driven Airflow Pipeline with SNS, SQS and Lambda.
• I am also reusing NGROK setup steps from the event drive pipeline article above.
• I am also reusing the Lambda function code from the same…