[GitHub] [airflow] jedcunningham commented on issue #11982: Scheduler not exiting after SIGTERM

[GitBox <gi...@apache.org>]

jedcunningham commented on issue #11982:
URL: https://github.com/apache/airflow/issues/11982#issuecomment-738344935


   We are using a `centos:8` based image (though it was happening with `centos:7` also). No entrypoint, a straight `airflow scheduler`. Yes, still happens in beta3.
   
   Another way to observer this is with `kubectl delete pod`, which sends a sigterm and (by default) if the container hasn't exited in 30s k8s sends a sigkill. So you can swap my "exec kill 1" with a naive delete pod and if the old pod sticks around for 30s you've ran across this issue. Airflow would never have exited (from what I've seen), k8s just comes along and forces it. e.g:
   `date; kubectl delete pods {scheduler_pod} --grace-period=120; date`
   (^ adjust the grace period as desired)
   (Also only reproduces without dumb-init - more on that later)
   
   With breeze:
   ```
   $ ./breeze kind-cluster start
   $ ./breeze kind-cluster deploy
   $ ./breeze kind-cluster shell
   $ kubectl get pods
   $ kubectl exec -i -t {scheduler_pod} bash
   // note: background then foreground so the pid is printed
   # airflow scheduler & fg
   
   // Now, in another shell
   $ ./breeze kind-cluster shell
   $ kubectl exec -i -t {scheduler_pod} bash
   # kill {pid from above}
   // note, it doesn't actually exit
   # kill {pid from above}
   // now it exits
   ```
   
   Now, dumb-init might be why you aren't able to replicate. By default, dumb-init forwards signals to ALL childen, not just the direct child:
   https://github.com/Yelp/dumb-init#session-behavior
   
   If sigterm is sent to them all, it exits cleanly:
   ```
   # dumb-init -- airflow scheduler & fg
   ```
   
   However, if you toggle `--single-child`, you still see the same behavior:
   ```
   # dumb-init --single-child -- airflow scheduler & fg
   ```
   
   Seems like airflow should cleanly exit with a sigterm to just the primary process.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org