Airflow DAG B runs many times after DAG A is completed for that day

DAG A has schedule '0 6 * * *'.

DAG B has schedule '*/5 * * * *'.

However, DAG B should only start running for that day once DAG A has completed for that day.

I've played around with SubDags and ExternalTaskSensor but haven't yet found a satisfactory solution and I'm sure I'm missing something good. Recommendations?

Edit: say DAG A is my ETL. DAG B has some tasks that query my database and require that data to be up-to-date. DAG B gets run throughout the day, but only once the ETL is completed.

I can see using ShortCircuitOperator, for example, and having the condition be "DAG A that ran today is completed." But how could I write this condition?

How was your experience with SubDAGs? Did anything not work as expected?
– Taylor Edmiston
Jun 29 at 21:33

I looked into SubDAGs because I mistakenly though "ah, so DAG B can be triggered once DAG A completes, and then I can have SubDAG C of DAG B that will run every 5 minutes once it's kicked off" ... but then I found that the SubDAG has to run on the same schedule as its parent DAG, so I hadn't actually gained anything.
– smartestblonde
Jul 2 at 1:36

2 Answers
2

This question is not an exact duplicate but is similar to another which already has 3 good answers: Scheduling dag runs in Airflow.

I recommend reading through all of them, but to summarize the info in the answers over there, there are several viable options for the use case of a DAG dependent upon another DAG:

If you can add more detail around the use case you're trying to accomplish, I could give more specific guidance as well.

So, looking into each of these, I see that they do work if I want to trigger one dag after another dag finishes. However, what I want is DAG B to be on a schedule that runs every 5 minutes, but only to run if DAG A is completed for that day. Each of these solutions, from what I can see, trigger just one run of DAG B -- am I missing something?
– smartestblonde
Jul 2 at 1:33

Use TriggerDagRunOperator for calling a DAG to run after another. Refer to this question. Afraid I cannot provide a satisfactory example as I have not used it yet.

TriggerDagRunOperator

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

搜尋此網誌

Fjhtyj