[GitHub] [airflow] getaaron commented on issue #31105: airflow-webserver crashes if log files are larger than webserver available RAM

["getaaron (via GitHub)" <gi...@apache.org>]

getaaron commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1605119355

   > Yeah, I thought of a way where we could maintain separate log_pos for different streams. This way we do not have to pull the whole file into memory just to get the final few lines. But I have some reservations about this with regard to whether the log order would change. I will look into this.
   
   If the separate log files are already sorted (I assume they are) then you can use a k-way merge to produce a sorted combined list without loading the whole thing into memory: https://en.m.wikipedia.org/wiki/K-way_merge_algorithm
   
   If they're not already sorted (unlikely) then you could sort them individually, then use the k-way merge sort.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org