[GitHub] [airflow] potiuk commented on issue #13591: Graceful termination does not work with apache chart running 1.10.14

[GitBox <gi...@apache.org>]

potiuk commented on issue #13591:
URL: https://github.com/apache/airflow/issues/13591#issuecomment-757720643


   I will take a look later this week. It also depends which command is used to run airflow components. You are.talking about the current master version of the 'chart' yeah ? No modification to the entrypoint or command ?
   
   The dumb init and tini are equivalent and they are indeed there to forward signals to the running processes (this is really useful when you have a bash script as entrypoint (if you have bash as direct entrypoint then it will not forward signals to it's children. There are two solutions to solve it:
   
   A) dumb init or tini as entrypoint 
   b) exec 'binary' at the end of the bash script (exec another bash won't work)
   
   Default entrypoint in prod image is dumb-init so it should propagate the signals properly, but as @xinbinhuang mentioned when you have celery worker it has a number of config options when you send a SIGTERM to it celery worker it will stop spawning new processes and wait for all the running tasks to terminate. So by definition the worker might take quite some time to exit. There is the termination grace period that controls how long it will take for the celery to wait for all processes to terminate before it will 'kill -9' and exits 'non gracefully'. 
   
   Also there is another gotcha - if you send SECOND SIGTERM to such celery worker while it is waiting for tasks, it will terminate all the processes with 'kill -9' and will exit immediately.
   
   So if you expect the worker to terminate immeditaely you might have observed actually wrong behaviour where someone sent more than one SIGTERM to those workers (I've seen such setups) - but this is a rather bad idea IMHO.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org