You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/03 01:24:17 UTC
[GitHub] [airflow] maxnathaniel opened a new pull request, #28685: Handle ConnectionReset exception in Executor cleanup
maxnathaniel opened a new pull request, #28685:
URL: https://github.com/apache/airflow/pull/28685
closes #28328
Added exception handling around Executor's cleanup block. Do let me know if there is a better way of handling the ConnectionReset exception.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #28685:
URL: https://github.com/apache/airflow/pull/28685#issuecomment-1369445533
Yeah. I don’t know if it’s needed though, if you only move the `kube_scheduler` part out of the block I’d be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maxnathaniel commented on pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
maxnathaniel commented on PR #28685:
URL: https://github.com/apache/airflow/pull/28685#issuecomment-1369437968
I'm not sure if it's necessary to separate the handling since the connection reset issue doesn't seem like it's specific to `_flush_task_queue` (my best guess based on the error logs).
I suppose the main reason why we would separate the handling is for `_flush_result_queue` to continue execution in the event that `_flush_task_queue` throws an exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #28685:
URL: https://github.com/apache/airflow/pull/28685#issuecomment-1369314264
The exception in the issue happened during `_flush_task_queue`. Do you anticipate the exception to also happen elsewhere during this procedure? If not, I think it’s probably better to only catch exceptions in that function instead (and maybe also `_flush_result_queue`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maxnathaniel commented on pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
maxnathaniel commented on PR #28685:
URL: https://github.com/apache/airflow/pull/28685#issuecomment-1369398793
Good point. I anticipate that the we will need to include `_flush_result_queue `, `self.task_queue.join()` and `self.result_queue.join()` as well since the exception was caused by the connection which was reset. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #28685:
URL: https://github.com/apache/airflow/pull/28685#issuecomment-1369408578
I wonder if we need to separate handling of the two queues
```python
try:
self._flush_task_queue()
self.task_queue.join() # Should be empty.
except ConnectionResetError:
self.log.exception(...)
try:
self._flush_result_queue()
self.result_queue.join()
except ConnectionResetError:
self.log.exception(...)
if self.kube_scheduler:
self.kube_scheduler.terminate()
self._manager.shutdown()
```
Or this is entirely irrelevant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28685: Handle ConnectionReset exception in Executor cleanup
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28685:
URL: https://github.com/apache/airflow/pull/28685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org