[I] Remote logging provider support for separate reader and (multiple) writer(s) [airflow]

["MaicoTimmerman (via GitHub)" <gi...@apache.org>]

MaicoTimmerman opened a new issue, #48874:
URL: https://github.com/apache/airflow/issues/48874

   ### Description
   
   The newly introduced `RemoteLogIO`, moves the abstraction for remote logging from the `logging.Handler` to purely `read` and `upload`, let's use this ticket to formalize the interface for remote logging solutions in Airflow 3.x.
   
   ### Use case/motivation
   
    Most handlers provided by providers implement `logging.Handler` have adopted this pattern. Our use-case is to ship logs to Kafka and read from ElasticSearch. We need to separate the reading and writing in order to keep control over the ingestion speed in a multi-tenant ES cluster.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remote logging provider support for separate reader and (multiple) writer(s) [airflow]

["MaicoTimmerman (via GitHub)" <gi...@apache.org>]

MaicoTimmerman commented on issue #48874:
URL: https://github.com/apache/airflow/issues/48874#issuecomment-2911598176

   Leaving a though; multiple writers could already be achieved in Airflow, by utilizing multiple logging handlers. It only poses a minor challenge in config management, as the connection is only passable using `logging.remote_log_conn_id` in most providers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org