[GitHub] [airflow] mik-laj commented on issue #15415: S3 remote logging not working for airflow server components

[GitBox <gi...@apache.org>]

mik-laj commented on issue #15415:
URL: https://github.com/apache/airflow/issues/15415#issuecomment-822864528


   Airflow handles some logs in a special way and unfortunately, it is not easy to unify this as our logs have different characteristics. 
   
   **Logs for tasks** are saved to files because they are small and we can upload them after completing the tasks. When they are sent to object storage it is much easier to read to them, but each time you add a new line, we need to get all the contents and upload a new file with the full log contents. Object storages have limited support for appending operations.
   On the other hand, logs for other components are an endless stream of data that cannot be stopped to send the content. For this reason, conventional tools fit better because they are optimized to handle such operations. 
   
   I don't think we can fix this problem, but we can update the documentation to better describe these project assumption.
   
   Related issue: https://github.com/apache/airflow/issues/10593


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org