You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/04 19:33:51 UTC

[GitHub] [airflow] vilozio opened a new issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

vilozio opened a new issue #18721:
URL: https://github.com/apache/airflow/issues/18721


   ### Apache Airflow version
   
   2.1.4 (latest released)
   
   ### Operating System
   
   Debian
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   DAG properties `on_failure_callback` and `on_success_callback` are simple object attributes, but there are another depended attributes that are set only during initialization of a DAG - `has_on_failure_callback` and `has_on_success_callback`.
   
   For example, If we leave `on_failure_callback` attribute empty on initialization and then assign a callback to it later for a DAG object, then the `has_on_failure_callback` will be left False and during a scheduled dag run the callback will be skipped.
   
   https://github.com/apache/airflow/blob/13a558d658c3a1f6df4b2ee5d894fee65dc103db/airflow/models/dagrun.py#L480
   
   ### What you expected to happen
   
   When we set `on_failure_callback` or `on_success_callback` attribute after DAG object initialization, `has_on_failure_callback` and `has_on_success_callback` attributes are changed respectfully. 
   
   ### How to reproduce
   
   Create a DAG object without `on_failure_callback` attribute, then separately assign a function to this attribute. A scheduled dag run will not call the callback function after a DAG failed.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-933791940


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934880860


   Using `@property` for `has_*_callback` should fix it @vilozio 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] vilozio commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
vilozio commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934432416


   Thanks @potiuk for the detailed answer.
   
   My use case is a creation of a dynamic DAG.
   
   I'm using dag-factory library https://github.com/ajbosco/dag-factory to create DAGs from YAML files and then I'm also adding enhances to the created DAGs (like `on_failure_callback`) after initialization, but before assigning it to globals of a module and before the execution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934434727


   Yes, your understanding is correct Jarek.
   
   So long as the callbacks are set in the "file" (which you are doing with the dag factory) then it is fine, so not correctly setting the `has_*` attributes is a bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934588466


   Might be related to #18746 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934464492


   > So long as the callbacks are set in the "file" (which you are doing with the dag factory) then it is fine, so not correctly setting the `has_*` attributes is a bug.
   
   Cool !- So go ahead @vilozio 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934374354


   I have not thought that through entirely but I think that can have undesireable consequences.
   
   The "has_succes/failure_callbcks" property is especially added to handle "Serialized DAG" case - rather than serializing/deserializing the calbacks, only the "has" properties are serialized. So those properties are needed. And also I cannot immeediately see why the callbacks could not be set afterwards - but still during parsing of the DAG. 
   
   On the other hand, it would be interesting to hear @vilozio what is WHEN you would like to set the callbacks. If you would like to set them during "execute" methods of some of the tasks, then it's almost for sure not going to happen due to security and performance reasons. People might be tempted to do so though, if we allow to override it in general. The fact that you add it at initialization only is "stronger" than "during parsing" and makes it pretty obvious you should not do it "after" the DAG has been parsed and serialized.
   
   I am just wondering if there is a reason why you would like to be able to set it during DAG parsing and still not be able to pass it in the __init__ method?  Because I would actually be tempted to turn the callbacks into read-only properties after initialization to avoid any temptation of trying to set it during initialization.
   
   @kaxil @ashb -> do I get it right? Any other concerns/comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-936184811


   > Might be related to #18746
   
   Shouldn't be, since `default_args` are "merged" in by the metaclass before `__init__()` is called, so it'll see them directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil edited a comment on issue #18721: Add a setter to on_failure_callback and on_success_callback DAG properties

Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #18721:
URL: https://github.com/apache/airflow/issues/18721#issuecomment-934880860


   Using `@property` for `has_*_callback` should fix it @vilozio 
   
   similar to https://github.com/apache/airflow/blob/767a4f5207f8fc6c3d8072fa780d84460d41fc7a/airflow/models/dag.py#L1032-L1034


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org