[GitHub] [airflow] potiuk commented on issue #16737: Scheduler memory creeping and plateauing

[GitBox <gi...@apache.org>]

potiuk commented on issue #16737:
URL: https://github.com/apache/airflow/issues/16737#issuecomment-917677177


   The problem is most likely connected with the use PageCache - see extensive discussion in #14924. 
   
   The growing memory observed was most likely just growing cache, which is normal in case of writing log files and not rotating the files afterwards. The log files are cached and the cache is never cleaned until there is enough memory or you reach the maximum cache size configured in the kernel.
   
   This can be checked by checking type of memory - if it's a "working_set" memory, then it's not good, but if this is "cache" memory then all is expected (and harmless) - see #14924 for details on how you can check it. Such cache memory will be released when the log files are rotated/deleted/zipped or when you clean the cache manually (again see the issue to see how).
   
   Even though it is harmless, we added an advise to kernel in #18054 to not store logs in PageCache in the first place, which should make your graphs look nicer if that's the case (even though it was not an issue in the first place).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org