You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/19 02:39:39 UTC

[GitHub] [airflow] sighingnow opened a new issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

sighingnow opened a new issue #17073:
URL: https://github.com/apache/airflow/issues/17073


   **Description**
   
   We propose to add such features to airflow, which will allow the users to define a hook at the DAG level, and been executed before the `pre_execution` and after the `post_execution`.
   
   TODO: we will propose detailed API change if such change is acceptable.
   
   **Use case / motivation**
   
   Airflow supports the operator to define their custom `pre_execution/post_execution` method, which will be exeucted before/after the execution of the operator, respectively.
   
   But is hard to inject such hooks to existing operators, without modifying the code. i.e., airfow doesn't support a global pre-hook/post-hook to run before/after execution of every operators. 
   
   Such kind of hooks could be used for preparing the required inputs and finalizing the produced outputs for every operators in the DAG, etc.
   
   **Are you willing to submit a PR?**
   
   <!--- We accept contributions! -->
   Yes, we would like to submit a PR if this feature request is acceptable.
   
   **Related Issues**
   
   None.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr edited a comment on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882191138


   A DAG-level solution feels limiting to me. While it sounds convenient, it is not actually very often a user would want the same hook to be executed for every task, so you’d probably end up with a lot of if-else in the function to handle different tasks in the end. A better approach to me would be to allow supplying the hook without subclassing, something like
   
   ```python
   def pre_exec(task: BaseOperator, context: Any) -> None:
       ...
   
   def different_pre_exec(task: BaseOperator, context: Any) -> None:
       ...
   
   with DAG(...) as dag:
       t1 = BashOperator(
           ...,
           before_exec_hook=pre_exec,
       )
       t2 = BashOperator(
           ...,
           before_exec_hook=pre_exec,
       )
       t3 = BashOperator(
           ...,
           before_exec_hook=different_pre_exec,
       )
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj edited a comment on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882202784


   @sighingnow What do you think about using ``default_args`` to define one common callback per DAG? Does it allow you to address your user case?
   https://github.com/apache/airflow/blob/ab915201068919b02e8e00e723f127715f71a12b/airflow/example_dags/example_bash_operator.py#L34


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882191138


   A DAG-level solution feels limiting to me. While it sounds convenient, it is not actually very often a user would want the same hook to be executed for every task, so you’d probably end up with a lot of if-else in the function to handle different tasks in the end. A better approach to me would be to allow supplying the hook without subclassing, something like
   
   ```python
   def pre_exec(task: BaseOperator, context: Any) -> None:
       ...
   
   def different_pre_exec(task: BaseOperator, context: Any) -> None:
       ...
   
   with DAG(...) as dag:
       t1 = BashOperator(
           ...,
           before_exec_hook=pre_exec,
       )
       t2 = BaseOperator(
           ...,
           before_exec_hook=pre_exec,
       )
       t3 = BaseOperator(
           ...,
           before_exec_hook=different_pre_exec,
       )
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj edited a comment on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882202784


   @sighingnow What do you think about using ``default_args``? Does it allow you to address your user case?
   https://github.com/apache/airflow/blob/ab915201068919b02e8e00e723f127715f71a12b/airflow/example_dags/example_bash_operator.py#L34


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] sighingnow commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
sighingnow commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882214251


   > Does it allow you to address your user case?
   
   Thanks for the information. I think that would work, it the default_args is available before executing the task. I need to do experiment on it and will repost it here once I have some results.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] sighingnow commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
sighingnow commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-889609376






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882188493


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882202784


   @sighingnow What do you think about using ``default_args ``? Does it allow you to address your user case?
   https://github.com/apache/airflow/blob/ab915201068919b02e8e00e723f127715f71a12b/airflow/example_dags/example_bash_operator.py#L34


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] sighingnow commented on issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
sighingnow commented on issue #17073:
URL: https://github.com/apache/airflow/issues/17073#issuecomment-882202002


   In our proposal the pre/post execution hooks only care about the linage information. Which commonly should be the same logic for every operator.
   
   Supplying with an extra optional arguments for every operator does work, but it means we need to update at every place where we define an instance of some operator. Supplying the hooks at the DAG level makes it possible by just modifying the `DAG(...)`, which would be less disturbing for end users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] sighingnow closed issue #17073: Support defining pre/post hook for operators in a DAG at the DAG level

Posted by GitBox <gi...@apache.org>.
sighingnow closed issue #17073:
URL: https://github.com/apache/airflow/issues/17073


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org