[GitHub] [pinot] walterddr commented on pull request #7679: add Metadata SPI interfaces and pinot-metadata module

[GitBox <gi...@apache.org>]

walterddr commented on pull request #7679:
URL: https://github.com/apache/pinot/pull/7679#issuecomment-957964708


   > Is it intended that in addition to logging an info/warn/error message, pinot also writes to a database? 
   Partially, yes. But currently there's no guarantee that the logs consist of all the metadata information and there's no way to register specific messages to be logged to external metadata service. 
   
   > A blocking write? Instead, why not try one of the following:
   For the SPI design, it doesn't necessarily have to be a blocking write. It is up to the SystemMetricsRegistry to spin up execution service to handle the `flush` but it is up to the user to define where to `collect`. (the example in the test might've been a bad one)
   
   
   > * An independent process that pulls messages from log
   As I mentioned log might not contain all data needed, also it could contain too much irrelevant info to parse out.
   
   > * A log pipeline that publishes asynchronously (light-weight) to a stream
   Yes, this is the basic intend of the SPI. In fact one implementation of the SPI can simply be a logging service (with schema) that writes to a separate file. However in this setting the external (light-weight) process that publish async to a stream will not be part of the SPI itself.
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org