dynamic dag generation

[Robin Edwards <ro...@goshift.com>]

Hello!

I have a DAG where the input size (rows) may grow or shrink significantly.

The first step (A) determines the size of the input set and groups into
batches of a pre-defined size.

The second step I want to generate a task per batch to perform an upload to
a third party API (google adwords) / computation.

The final step is a sensor which waits for the status of the batch to be
completed and then a final task.

Thoughts so far:

- I don't necessarily need all tasks to execute in parallel I just want to
be able to control the number that do through Pools
- I could potentially calculate the batch size and number of tasks required
at DAG compile time but this would make my DAG loading very slow (as I will
have lots of DAGs doing this)
- Is changing the number of tasks in a DAG dynamically going to screw up
airflow?
- I found this https://stackoverflow.com/a/51977800 but it feels a bit of a
hack.
- I could trigger multiple dagruns but this makes it harder to visualise
and trace through the UI

Or am i approaching this problem in the wrong way?

Thanks for your help,

Rob