You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/26 22:08:13 UTC
[GitHub] [airflow] potiuk opened a new pull request, #23274: Remove custom signal handling in Triggerer
potiuk opened a new pull request, #23274:
URL: https://github.com/apache/airflow/pull/23274
There is a bug in CPython (fixed in March 2022 but not yet released) that
makes async.io handle SIGTERM improperly by using async unsafe
functions and hanging the triggerer receive SIGPIPE while handling
SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
we should rather rely on standard handling of the signals rather than
adding our own signal handlers. Seems that even if our signal handler
just run exit(0) - it caused a race condition that led to the hanging.
Fixes: #19260
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #23274:
URL: https://github.com/apache/airflow/pull/23274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanelk commented on pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
tanelk commented on PR #23274:
URL: https://github.com/apache/airflow/pull/23274#issuecomment-1148818199
@potiuk It seems that after this we do not change the TriggererJob state to success any more. They stay in running:
There is also register_signals method in the triggerer job, but it is never called.
https://github.com/apache/airflow/blob/717a7588bc8170363fea5cb75f17efcf68689619/airflow/jobs/triggerer_job.py#L67-L70
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23274:
URL: https://github.com/apache/airflow/pull/23274#issuecomment-1110304559
I also found that the standalone problem with hanging was already reported in #19260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23274:
URL: https://github.com/apache/airflow/pull/23274#issuecomment-1110302515
CC: @andrewgodwin @ashb @dstandish -> I was able to reproduce the Ctrl-C problem and it's gone after I removed the custom signal handling in Triggerer, so it looks like the hypothesis of the async.io bug from https://bugs.python.org/issue39622 https://github.com/python/cpython/issues/83803 seems even more plausible.
Pls. take a look and see if my Hypothesis from https://github.com/apache/airflow/pull/23271#issuecomment-1110260794 looks sound and maybe we can just fix it permanently also for production.
I believe our custom signal handling of SIGINT and SIGTERM in Triggerer (which then would simply run sys.exit(0) ) is not really needed (default handling of both signals terminates the process eventually). I left SIGQUIT handling though for diagnostics (And QUIT is rarely used anyway for anything else).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23274:
URL: https://github.com/apache/airflow/pull/23274#issuecomment-1152937848
Thanks @tanelk - reverting it for 2.3.3 then and reopened the original issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #23274: Remove custom signal handling in Triggerer
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23274:
URL: https://github.com/apache/airflow/pull/23274#issuecomment-1110939570
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org