[GitHub] [airflow] potiuk edited a comment on pull request #17576: Add pre/post execution hooks

[GitBox <gi...@apache.org>]

potiuk edited a comment on pull request #17576:
URL: https://github.com/apache/airflow/pull/17576#issuecomment-916379253


   One of the good cases is to distinguish between DEV/PROD environments. 
   
   For example some of the operators  could be skipped on DEV environments and you might want to add single method throwing "Ski[p" exception in the DEV environment (for costly operations) 
   
   This is mostly a feature that promotes good engineering practices such as being able to run the same code on different environments, wtihout changing the DAGs or adding artifficial logic in the DAG structure, to handle different execution environments. 
   
   I think this is the main case (often raised by our users in Slack). If we do not have hooks, the users would have to develop custom operators of different kind to handle different behaviours. Having the configurable hooks you can deploy the same DAGs for different environments without adding complex logic or having to add custom operators.
   
   Many of our users (and DAG writers) are Data scientists. You could potentially have the same effect by extending N operators and adding pre/post hooks there but for those users classes and objects are "alien" - they think functional and adding pre/post hooks is so much easier, especially that you can much more easliy have one common function that you pass to a number of different, completley unrelated operators. This is a classic "cross-cutting-concern" implementation IMHO.
   
   This is at least what has been my motivation when guiding and approving this one.
   
   Any other idea how to approach it? I am not against other solutions, and I would like to hear what anti-patterns it promotes (in the context I described). Could you tell @ash what antipattern is in play here, as I am not sure  I see it ?
   
   Of course if it somehow makes other parts (non-completed AIP's especially) much more difficult to implement. we can change the way it is implemented, but I think the cabability of making "cross-cutting-concern" - for example tied to an environment where the DAG is executed is a powerful feature that might make the deployment and testing DAGs at scale much easier. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org