You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/01 16:20:25 UTC
[GitHub] [airflow] potiuk opened a new pull request #19935: Fix race condition with killing processor mnager
potiuk opened a new pull request #19935:
URL: https://github.com/apache/airflow/pull/19935
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19935:
URL: https://github.com/apache/airflow/pull/19935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#discussion_r761378266
##########
File path: airflow/utils/process_utils.py
##########
@@ -55,7 +55,12 @@ def reap_process_group(
sig (SIGTERM) to the process group of pid. If any process is alive after timeout
a SIGKILL will be send.
- :param pgid: process group id to kill
+ :param process_group_id: process group id to kill. This process group id is retrieved earlier from
+ the process id of the process that started that process group. The process that wants to
+ create the group should run `os.setpgid(0, 0)` as the first command it executes which will
+ set group id = process_id. Effectively the process that is the "root" of the group has
+ pid = gid and all other processes in the group have different pids but the same gid (equal
+ the pid of the root process)
Review comment:
I meant what you explained better!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#discussion_r760997495
##########
File path: airflow/utils/process_utils.py
##########
@@ -68,36 +73,51 @@ def on_terminate(p):
def signal_procs(sig):
try:
- os.killpg(pgid, sig)
+ logger.info("Sending the signal %s to group %s", sig, process_group_id)
+ os.killpg(process_group_id, sig)
except OSError as err:
# If operation not permitted error is thrown due to run_as_user,
# use sudo -n(--non-interactive) to kill the process
if err.errno == errno.EPERM:
subprocess.check_call(
- ["sudo", "-n", "kill", "-" + str(int(sig))] + [str(p.pid) for p in children]
+ ["sudo", "-n", "kill", "-" + str(int(sig))]
+ + [str(p.pid) for p in all_processes_in_the_group]
)
+ elif err.errno == errno.ESRCH:
+ # There is a rare condition that the process has not managed yet to change it's process
+ # group. In this case os.killpg fails with ESRCH error
+ # So we additionally send a kill signal to the process itself.
+ logger.info(
+ "Sending the signal %s to process %s as process group is missing.", sig, process_group_id
+ )
+ os.kill(process_group_id, sig)
Review comment:
This could, again, fail with EPERM, so we might need to refactor this to handle the sudo case better :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#discussion_r761382587
##########
File path: airflow/utils/process_utils.py
##########
@@ -68,36 +73,51 @@ def on_terminate(p):
def signal_procs(sig):
try:
- os.killpg(pgid, sig)
+ logger.info("Sending the signal %s to group %s", sig, process_group_id)
+ os.killpg(process_group_id, sig)
except OSError as err:
# If operation not permitted error is thrown due to run_as_user,
# use sudo -n(--non-interactive) to kill the process
if err.errno == errno.EPERM:
subprocess.check_call(
- ["sudo", "-n", "kill", "-" + str(int(sig))] + [str(p.pid) for p in children]
+ ["sudo", "-n", "kill", "-" + str(int(sig))]
+ + [str(p.pid) for p in all_processes_in_the_group]
)
+ elif err.errno == errno.ESRCH:
+ # There is a rare condition that the process has not managed yet to change it's process
+ # group. In this case os.killpg fails with ESRCH error
+ # So we additionally send a kill signal to the process itself.
+ logger.info(
+ "Sending the signal %s to process %s as process group is missing.", sig, process_group_id
+ )
+ os.kill(process_group_id, sig)
Review comment:
Fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#issuecomment-985092064
Woohooo!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#discussion_r760996802
##########
File path: airflow/utils/process_utils.py
##########
@@ -55,7 +55,12 @@ def reap_process_group(
sig (SIGTERM) to the process group of pid. If any process is alive after timeout
a SIGKILL will be send.
- :param pgid: process group id to kill
+ :param process_group_id: process group id to kill. This process group id is retrieved earlier from
+ the process id of the process that started that process group. The process that wants to
+ create the group should run `os.setpgid(0, 0)` as the first command it executes which will
+ set group id = process_id. Effectively the process that is the "root" of the group has
+ pid = gid and all other processes in the group have different pids but the same gid (equal
+ the pid of the root process)
Review comment:
```suggestion
:param process_group_id: process group id to kill. This process group id is retrieved earlier from
the process id of the process that started that process group.
The process that wants to
create the group should run `os.setpgid(0, 0)` as the first command it executes which will
set group id = process_id. Effectively the process that is the "root" of the group has
pid = gid and all other processes in the group have different pids but the same gid (equal
the pid of the root process)
```
I'm also not quite sure what you mean by "This process group id is retrieved earlier from the process id of the process that started that process group"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#issuecomment-983858789
Looks really good with all tests green !!! :). Cannot wait to merge it and rebase the last 'stability' one :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#discussion_r761377814
##########
File path: airflow/utils/process_utils.py
##########
@@ -68,36 +73,51 @@ def on_terminate(p):
def signal_procs(sig):
try:
- os.killpg(pgid, sig)
+ logger.info("Sending the signal %s to group %s", sig, process_group_id)
+ os.killpg(process_group_id, sig)
except OSError as err:
# If operation not permitted error is thrown due to run_as_user,
# use sudo -n(--non-interactive) to kill the process
if err.errno == errno.EPERM:
subprocess.check_call(
- ["sudo", "-n", "kill", "-" + str(int(sig))] + [str(p.pid) for p in children]
+ ["sudo", "-n", "kill", "-" + str(int(sig))]
+ + [str(p.pid) for p in all_processes_in_the_group]
)
+ elif err.errno == errno.ESRCH:
+ # There is a rare condition that the process has not managed yet to change it's process
+ # group. In this case os.killpg fails with ESRCH error
+ # So we additionally send a kill signal to the process itself.
+ logger.info(
+ "Sending the signal %s to process %s as process group is missing.", sig, process_group_id
+ )
+ os.kill(process_group_id, sig)
Review comment:
Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19935: Fix race condition when starting DagProcessorAgent
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19935:
URL: https://github.com/apache/airflow/pull/19935#issuecomment-985089971
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org