[GitHub] dlg99 opened a new issue #1409: Server-side backpressure

[GitBox <gi...@apache.org>]

dlg99 opened a new issue #1409: Server-side backpressure
URL: https://github.com/apache/bookkeeper/issues/1409
 
 
   **FEATURE REQUEST**
   
   1. Please describe the feature you are requesting.
   
   It is possible for client(s) to overload bookie with the requests.
   Bookie should stop accepting requests if it cannot process them anymore.
   
   Without backpressure, bookie:
   
   Scenario 1:
   Bookie receives add entry requests and pushes them to downstream processing. 
   These requests end up accumulated in journal queue. 
   Bookie runs out of memory, goes into super long GC or just drags on processing requests with extremely high latency trying to handle all queued requests.
   
   Scenario 2:
   Bookie receives more requests than it can handle, memtable flush is super slow, bookie tries to slow things down (sleep 1ms as a throttling attempt in memtable)
   Netty keeps on autoreading requests from the network into internal buffers
   bookie runs out of memory, goes into super long GC or just drags on processing requests with >> 1sec latency trying to handle all queued requests.
   
   Scenario 3:
   Bookie receives a lot of read requests.
   Bookie manages to actually read data on time
   Bookie cannot send all read data back to the client because of NIC being maxed out, client is not responding etc.
   Bookie runs out of memory, goes into super long GC or just drags on processing requests with >> 1sec latency trying to handle all queued requests.
   (yes, I noticed an option to shed read requests)
   
   2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   
   must have
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services