[jira] [Created] (AIRFLOW-5109) reap_process_group() can still result in uncaught ProcessLookupErrors / OSErrors

["Hunter Blanks (JIRA)" <ji...@apache.org>]

Hunter Blanks created AIRFLOW-5109:
--------------------------------------

             Summary: reap_process_group() can still result in uncaught ProcessLookupErrors / OSErrors
                 Key: AIRFLOW-5109
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5109
             Project: Apache Airflow
          Issue Type: Bug
          Components: utils
    Affects Versions: 1.10.5
         Environment: Found in production using current python3.6-slim-stretch Docker image and airflow 1.10.3, also present on current master branch.
            Reporter: Hunter Blanks


{{airflow.utils.helpers.reap_process_group()}} can result in uncaught ProcessLookupError's due to race conditions in calls to Python's {{os}} module.

{noformat}
ProcessLookupError: [Errno 3] No such process
  File "airflow", line 32, in <module>
    args.func(args)
  File "airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "airflow/bin/cli.py", line 523, in run
    _run(args, dag, ti)
  File "airflow/bin/cli.py", line 437, in _run
    run_job.run()
  File "airflow/jobs.py", line 209, in run
    self._execute()
  File "airflow/jobs.py", line 2568, in _execute
    self.heartbeat()
  File "airflow/jobs.py", line 192, in heartbeat
    self.heartbeat_callback(session=session)
  File "airflow/utils/db.py", line 69, in wrapper
    return func(*args, **kwargs)
  File "airflow/jobs.py", line 2632, in heartbeat_callback
    self.task_runner.terminate()
  File "airflow/task/task_runner/standard_task_runner.py", line 41, in terminate
    reap_process_group(self.process.pid, self.log)
  File "airflow/utils/helpers.py", line 290, in reap_process_group
    os.killpg(os.getpgid(pid), signal.SIGKILL)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)