[GitHub] [airflow] potiuk commented on issue #15799: celery worker timed out with rabbitmq

[GitBox <gi...@apache.org>]

potiuk commented on issue #15799:
URL: https://github.com/apache/airflow/issues/15799#issuecomment-1295152007

   @theophanevie Seems this might be simply problem with configuration of your rabbitmq not matching the load/characteristics of your traffic (which might be either too low and the problem is with idle connections or too high when cellery workers/rabbitmq cannot cope with the load.
   
   Can you please try the solutions decribed here:
   
   * https://groups.google.com/g/celery-users/c/5dPhSfF5APQ
   
   and possibly here:
   
   * https://stackoverflow.com/questions/52072305/celery-worker-stops-after-being-idle-for-a-few-hours ? 
   
   It seems that might at least reduce the frequency of similar problems
   
   Also it woudl be great if you could describe what kind of setup, sizing, workload characteristics you have and tell us more about  how frequently, in what circumstances it happens. Without having that knowledge  - and just seeing "timeout" message, it next to impossible even to guide you in the direction of how you can solve your problem - which is again, most likely something that needs reconfiguring your rabbitmq. 
   
   Typically this is done by those who manage their deployments via looking at the load and reconfiguring their components as needed - there are no specific quidelines from Airflow on how to do it as every installation has its own set of characteristics and needs indivdual tuning. So you absolutely need to make such an exercise on your load and performance and try to correlate that with the errors you see - before you attempt to fix it.
   
   If you provide the description and circumstances and some kind of result of investigation of what triggers the problem we here might be able to help to tune the setting - but without that, I am afraid it will be difficiult to help to solve your problem.
   
   And BTW. I am converting that into a discussion - because this is most likely question of rabbitmq configuration not Airflow itsefl.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org