[GitHub] [airflow] jedcunningham commented on pull request #25121: Add "Optimizing" chapter to dynamic-dags section

[GitBox <gi...@apache.org>]

jedcunningham commented on PR #25121:
URL: https://github.com/apache/airflow/pull/25121#issuecomment-1191634372

   The new paragraph looks good, and it does give us something to point at to say "we told you it was experimental", I'm not sure it changes much.
   
   I'm not exactly sure how to articulate my thoughts, but I'll try.
   
   This feels to me like we are helping hide the fact people didn't architect their DAGs properly. If you've hit the point where this is necessary, then you've already strayed out of the norm. Putting out a complex chuck of code that'll be copy pasted around, getting out of date, what have you is just adding an extra burden to maintenance and support. I can totally imagine down the road an upgrade breaking an early version of this and us getting hard to diagnose bug reports because of it. Worse, if the users DAGs are big/complex enough to need this, they likely can't/won't share it as a reproduction so that adds an extra burden.
   
   There are other ways to DRY up a DAG than to have them all come from a single loop 🤷‍♂️.
   
   I have no problem with folks opting in to this, knowing (hopefully) exactly the risks that come with it. However once we toss it in the official docs (even with a disclaimer paragraph), it becomes "officially blessed" to an extent, and I'm not sure we should do that for this iteration.
   
   tldr: very clever solution, but too esoteric/brittle to be officially documented for end users imo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org