You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/03 15:05:03 UTC

[GitHub] [airflow] MammutMKII opened a new issue #19384: Add retries to LivyOperator polling / LivyHook

MammutMKII opened a new issue #19384:
URL: https://github.com/apache/airflow/issues/19384


   ### Description
   
   Add an optional retry loop to LivyOperator.poll_for_termination() or LivyHook.get_batch_state() to improve resiliency against temporary errors. The retry counter should reset with successful requests.
   
   ### Use case/motivation
   
   1. Using LivyOperator, we run a Spark Streaming job in a cluster behind Knox with LDAP authentication.
   2. While the streaming job is running, LivyOperator keeps polling for termination.
   3. In our case, the LDAP service might be unavailable for a few of the polling requests per day, resulting in Knox returning an error.
   4. LivyOperator marks the task as failed even though the streaming job should still be running, as subsequent polling requests might have revealed.
   5. We would like LivyOperator/LivyHook to send a number of retries in order to overcome those brief availability issues.
   
   Workarounds we considered:
   - increase polling interval to reduce the chance of running into an error. For reference, we are currently using an interval of 10s
   - use BaseOperator retries to start a new job, only send notification email for the final failure. But this would start a new job unnecessarily
   - activate knox authentication caching to decrease the chance of errors substantially, but it was causing issues not related to Airflow
   
   ### Related issues
   
   No related issues were found
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] MammutMKII commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
MammutMKII commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-994678362


   Sorry for the late response.
   @potiuk I thought about fixing the issue through customization within the company and then submitting the code here as a MR. But getting a local setup and getting behind the test setup and contributing doesn't seem possible at the moment.
   So far I didn't get to do the custom implementation though and likely won't be soon.
   
   
   @kazanzhy Yes, but technically, from the perspective of Airflow, it's a 24h long batch operation with a 24h schedule. Specifically, every 24h we start the Spark application which:
   - first deletes a flag file
   - waits some time
   - writes the flag file
   - starts the Spark stream(s)
   - waits until the flag file disappears
   - stops the Spark stream(s)
   - deletes the flag file
   
   So in practice, every 24h we will start a DAG run which replaces the previous DAG run, with a short downtime for the stream and a short period where we have two DAG runs and Spark applications running at the same time.
   
   It's probably not good practice as it's not the intended use of Airflow and the stream uptime isn't 100%. Also, the cluster needs to be able to allocate the resources for two Spark applications for the transition period. Although I think the last point could be eliminated if we moved deleting the flag file to some upstream task. Airflow itself also dedicates some resources to running the task all day, but with the retries and notification emails I'd consider it worthwhile.
   
   So far we only had issues with the reported issues and our rather small cluster size. On the upside, we have all of our use cases in one place.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kazanzhy commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
kazanzhy commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-989998325


   I wonder the streaming jobs run using Airflow are good practice.
   So you have the DAGs that continuously running? 
   @MammutMKII 
   
   
   I'm can try to implement retries for LivyOperator. Could you assign it to me? 
   @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] utkarsharma2 edited a comment on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
utkarsharma2 edited a comment on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-1040757169


   @potiuk, we can implement retries in multiple ways, right now I have chosen option 1:
   
   1. We can use HttpHook’s run_with_advanced_retry method which has retry build into it via tenacity, and let user have complete control of how the retry work via choosing options by passing _retry_args_ parameter, probably good for advance users.
   2. Using same code as above option but providing default values for _retry_args_, which make sense for most cases and if power user wants to overwrite, they can.
   3. We can use tenacity’s _Retrying_ method to modify call to hook.get_batch_state() at [line](https://github.com/apache/airflow/pull/21550/files#diff-b8586f3007fabbc894632e59111527536b03a2f576bda13bbcec5df9d3c1338bR153) and pass retry_args to the retryer decorated function.
   Example - 
   `retryer = Retrying(retry_args)`
   `retryer(hook.get_batch_state, batch_id)`
   
   Please let me know your opinion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] utkarsharma2 commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
utkarsharma2 commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-1040757169


   @potiuk, we can implement retries in multiple ways, right now I have chosen option 1:
   
   1. We can use HttpHook’s run_with_advanced_retry method which has retry build into it via tenacity, and let user have complete control of how the retry work via choosing options by passing _retry_args_ parameter, probably good for advance users.
   2. Using same code as above option but providing default values for _retry_args_, which make sense of all users and if power user wants to overwrite, they can.
   3. We can use tenacity’s _Retrying_ method to modify call to hook.get_batch_state() at [line](https://github.com/apache/airflow/pull/21550/files#diff-b8586f3007fabbc894632e59111527536b03a2f576bda13bbcec5df9d3c1338bR153) and pass retry_args to the retryer decorated function.
   Example - 
   `retryer = Retrying(retry_args)`
   `retryer(hook.get_batch_state, batch_id)`
   
   Please let me know your opinion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-962702654


   Would you maybe like to implement it  @MammutMKII ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19384:
URL: https://github.com/apache/airflow/issues/19384


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-959367162






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kazanzhy edited a comment on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-989998325


   I wonder the streaming jobs run using Airflow are good practice.
   So you have the DAGs that continuously running? 
   @MammutMKII 
   
   
   I'm can try to implement retries for LivyOperator. I think it may be helpful for long-running batch operations.
   Could you assign it to me? 
   @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-959367162


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19384: Add retries to LivyOperator polling / LivyHook

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19384:
URL: https://github.com/apache/airflow/issues/19384#issuecomment-959367162


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org