[GitHub] [airflow] gdevanla edited a comment on issue #15938: celery_executor becomes stuck if child process receives signal before reset_signals is called

[GitBox <gi...@apache.org>]

gdevanla edited a comment on issue #15938:
URL: https://github.com/apache/airflow/issues/15938#issuecomment-845671073


   I just upgraded to 2.0.2 and this happens almost within 1 minute of scheduler restart.  We are on Python 3.8 and using RabbitMq/Celery.
   
   ```
   [2021-05-21 05:40:15,996] {channel.py:446} DEBUG - Channel open
   [2021-05-21 05:40:16,000] {channel.py:105} DEBUG - using channel_id: 1
   [2021-05-21 05:40:16,005] {channel.py:446} DEBUG - Channel open
   [[2021-05-21 05:40:16,053] {2021-05-21 05:40:16,053scheduler_job.py:] {746} INFOscheduler_job.py: - Exiting gracefully upon receiving signal 15746} INFO
    - Exiting gracefully upon receiving signal 15
   [2021-05-21 05:40:16,060] {scheduler_job.py:746} INFO - Exiting gracefully upon receiving signal 15
   [2021-05-21 05:40:16,067] {scheduler_job.py:746} INFO - Exiting gracefully upon receiving signal 15
   [2021-05-21 05:40:16,076] {scheduler_job.py:746} INFO - Exiting gracefully upon receiving signal 15
   [2021-05-21 05:43:34,715] {serialized_dag.py:197} DEBUG - Deleting Serialized DAGs (for which DAG files are deleted) from serialized_dag table 
   [2021-05-21 05:43:34,716] {dag.py:2213} DEBUG - Deactivating DAGs (for which DAG files are deleted) from dag table 
   [2021-05-21 05:43:34,723] {dagcode.py:135} DEBUG - Deleting code from dag_code table 
   [2021-05-21 05:43:48,746] {settings.py:292} DEBUG - Disposing DB connection pool (PID 6758)
   [2021-05-21 05:43:48,760] {scheduler_job.py:310} DEBUG - Waiting for <ForkProcess name='DagFileProcessor1-Process' pid=6758 parent=1204 stopped exitcode=0>
   ```
   
   And here is the corresponding logs in the `error` file: (I am not sure if this is related, but just thought I would add this if it could be useful)
   
   ```
   Process ForkPoolWorker-622:
   Traceback (most recent call last):
     File "/home/foo/py_src/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/home/foo/py_src/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/home/foo/py_src/lib/python3.8/multiprocessing/pool.py", line 114, in worker
       task = get()
     File "/home/foo/py_src/lib/python3.8/multiprocessing/queues.py", line 355, in get
       with self._rlock:
     File "/home/foo/py_src/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
       return self._semlock.__enter__()
     File "/home/foo/rapc/.env38/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 748, in _exit_gracefully
       self.processor_agent.end()
     File "/home/foo/rapc/.env38/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 458, in end
       self._process.join(timeout=1.0)
     File "/home/foo/py_src/lib/python3.8/multiprocessing/process.py", line 147, in join
       assert self._parent_pid == os.getpid(), 'can only join a child process'
   AssertionError: can only join a child process
   ```
   
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org