[GitHub] justinborromeo edited a comment on issue #7075: [PROPOSAL] 7057 Add a Configurable Event Throttling Policy to Asynchronous Emitters

[GitBox <gi...@apache.org>]

justinborromeo edited a comment on issue #7075: [PROPOSAL] 7057 Add a Configurable Event Throttling Policy to Asynchronous Emitters
URL: https://github.com/apache/incubator-druid/issues/7075#issuecomment-464173080
 
 
   I'm looking at the Kafka metrics emitter and I'm kinda stumped about a design decision.  The way it's currently implemented, it follows the pattern of all the other emitters where events are pushed to some queue then some thread dequeues events and does something with them.  In this case, the thread sends the event to Kafka using the KafkaProducer.  From what I understand, the Kafka producer has its own memory-bounded buffer where records are batched before actually being sent.  From the [javadoc](https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html):
   
   > The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster.
   
   If this is the case, is there any benefit of having the first queue if it's being buffered on the same machine at a later stage?  The only possibility I can think of is that the KafkaProducer#send() call blocks if the producer buffer is full -> however, if we set `max.block.ms` to 0, we should be able to get the same behaviour where records fast-fail upon a full buffer.  
   
   If the first queue isn't needed, can it be removed?  If so, `KafkaEmitter` wouldn't really fit into this Queue-based emitter pattern.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org