[GitHub] leventov opened a new issue #7057: Make asynchronous emitters have the same (configurable?) policy when event queue is full

[GitBox <gi...@apache.org>]

leventov opened a new issue #7057: Make asynchronous emitters have the same (configurable?) policy when event queue is full
URL: https://github.com/apache/incubator-druid/issues/7057
 
 
   This issue narrows the idea of #7037.
   
   There are many emitters (at least: `AmbariMetricsEmitter`, `GraphiteEmitter`, `StatsDEmitter` and `KafkaEmitter` and `HttpPostEmitter`, I haven't checked other) that use the same producer-consumer pattern for asynchronous emit: `emit()` pushes the event to some queue (one of the queues), and there is an asynchronous executor that retrieves events from the queue and sends them over network to emit.
   
   `AmbariMetricsEmitter` and `GraphiteEmitter` use the same policy when the queue is full (they log a warning). But `StatsDEmitter` apparently silently discards new events when the queue is full (see `NonBlockingStatsDClient` code). `KafkaEmitter` discards new events, but increments "lost events" counts. `HttpPostEmitter` packs events in batches and drops the oldest batch when overwhelmed, simultaneously logging that (see `HttpPostEmitter.limitBuffersToEmitSize()` and `limitFailedBuffersSize()`).
   
   I think all emitters should be similar in this regard. Probably backpressure policy should be configurable.
   
   Related to #2868.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org