[GitHub] nicmichael opened a new pull request #1504: Issue #1489: Better Prevent Read Outliers during short-term Bookie Slow-Down

[GitBox <gi...@apache.org>]

nicmichael opened a new pull request #1504: Issue #1489: Better Prevent Read Outliers during short-term Bookie Slow-Down
URL: https://github.com/apache/bookkeeper/pull/1504
 
 
   ### Motivation
   
   Bookies can temporarily be slow for a large number of reasons, often for just a brief time of few milliseconds to seconds such as during Java Garbage Collection or EntryLog compaction. For writes, latencies of individual bookies are masked by acknowledging the client after a quorum of bookies have replied. However for reads, we don't have any equivalent feature to mask short-term latencies of individual bookies yet (in case of SequenceReadRequests). This PR implements such a feature by reordering reads to prefer bookies with a high probability of being fast over bookies that are potentially slow.
   
   ### Changes
   This change implements a configurable reordering of read requests in Bokkeeper client based on the number of pending requests to each bookie that could service the request. The intention is to mask the latency of one bookie by directing a read request to another bookie that could potentially service the request faster. This should help prevent read time outliers due to bookies that temporarily are responsing slow, for example due to Java garbage collection, compaction, or any other kind of hickup. Unlike the implementation for Issue #709, this algorithm quickly reacts to both an or decrease increase in queue length of a bookie relative to others, and allows to redirect requests long before they would hit the speculativeReadTimeout. Once the problem is resolved (e.g. Java GC finished), it will quickly direct requests to the previously "slow" bookie as its queue length decreases.
   
   Reordering of reads is based on a threshold of relative queue length to other bookies. Setting the threshold very low will more frequently reorder the read set and potentially result in better latency, but will also reduce data affinity of reads. Reads send to other than the preferred bookie have a low chance to be served from file system cache on that bookie, and will likely result in a physical read. Small thresholds therefore shuffle read requests more among bookies and may lead to reduced file system cache reach and increased physical reads on disks. A larger timeout will maintain data affinity and avoid above problems, but only kick in once a bookie has built-up a considerable queue of requests. It therefore masks only slightly larger outliers, and leads to overall better efficiency.
   
   Master Issue: #1489 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services