[GitHub] [airflow] mtraynham commented on issue #13824: Cloudwatch Integration: SIGTERM/SIGKILL Sent Following DAG Completion, Causing Errors in Worker Logs

[GitBox <gi...@apache.org>]

mtraynham commented on issue #13824:
URL: https://github.com/apache/airflow/issues/13824#issuecomment-766093578


   One thing to note with this issue, I believe this is preventing workers from also consuming more work via a 60 second sleep at the end of every task.  We noticed that there was a ~30-40 second delay between 2 workers running tasks, as if all workers were busy doing something.  It seemed work had been queued, but was not being performed.
   
   After looking at the logs within for a given task in CloudWatch (which has the added benefit of including times in the log), there's a 60 sleep after SIGTERM to when the process is finally killed with SIGKILL.
   
   ```
   2021-01-23T15:00:02.682Z | Marking task as SUCCESS. dag_id=xxxxxxxx, task_id=xxxxxxx, execution_date=20210123T140000, start_date=20210123T150001, end_date=20210123T150002
   -- | --
     | 2021-01-23T15:00:02.715Z | 0 downstream tasks scheduled from follow-on schedule check
     | 2021-01-23T15:00:06.606Z | State of this instance has been externally set to success. Terminating instance.
     | 2021-01-23T15:00:06.607Z | Sending Signals.SIGTERM to GPID 155
     | 2021-01-23T15:01:06.617Z | process psutil.Process(pid=155, name='airflow task ru', status='sleeping', started='15:00:01') did not respond to SIGTERM. Trying SIGKILL
     | 2021-01-23T15:01:06.625Z | Process psutil.Process(pid=155, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='15:00:01') (155) terminated with exit code Negsignal.SIGKILL
     | 2021-01-23T15:01:06.625Z | Task exited with return code Negsignal.SIGKILL
   
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org