Re: [I] The Celery Worker Pods terminated and lost task logs when scale down happened during HPA autoscaling [airflow]

["Miretpl (via GitHub)" <gi...@apache.org>]

Miretpl commented on issue #41163:
URL: https://github.com/apache/airflow/issues/41163#issuecomment-5591226711

   My 2c:
   
   This issue is not really connected to the Helm Chart itself, but rather to how remote logging works and the Celery provider in the Kubernetes setup (this is why HPA doesn't really matter here). In any case, when the Celery worker is killed, the default (and basically the only supported) behaviour is for the worker to wait for all tasks to finish and then shut down. In a Kubernetes setup, after the SIGTERM, we have `terminationGracePeriod`, after which we have `SIGKILL`. The reason why there are no logs is the fact that Celery waits for tasks to finish, and logs are uploaded only after a task is finished. One way of fixing it would be to manually propagate the `SIGTERM` signal from Celery to the Task Supervisers to let them know that they have to kill the task (change in the Celery provider). The other way would be to change how remote logging works from "logging after finish" to "logging in the stream to the remote location" (change in Google, Azure, etc. providers with logging ha
 ndlers). Both of these have some pros and cons, like:
   1. Celery - requires synchronisation between when we will send the `SIGTERM` signal from the main process to task supervisors with pod `terminationGracePeriod` time
   2. Logging - moving from one upload to streaming can, e.g. increase cost, network traffic, etc., and we should be careful about it, as with scale, it might have a huge impact
   
   Also, the logging change would require a bit of architectural changes in how the upload works, as currently (as far as I remember) it is invoked in the `close` methods of the task handlers. Next to it, with the streaming approach, is the fact that if we had streaming, we could skip creating and logging to the file and copying it to the remote location - we could just do it directly.
   
   There is a chance that there are more ways of fixing this issue - none of them is on top of my mind now. I would leave the decision on how to solve it to the person who will propose a PR. Both log streaming and the Celery change can coexist.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org