[2/2] incubator-kudu git commit: java: use truncated randomized exponential backoff for retries

[to...@apache.org]

java: use truncated randomized exponential backoff for retries

This changes the Java client to use a randomized exponential backoff,
starting at 1ms, and going up to 4096ms, rather than the previous linear
backoff of (500ms * retry_count). The old backoff was far too
aggressive, and resulted in the Java client regularly having high
percentile latencies of 500 or 1000ms even when there was spare capacity
in the system.

I tested this on a 70 node cluster using a read-only YCSB workload, and
the 99.99th percentile latency dropped from >1s to 390ms.  Combined with
further in-flight changes, the 99.99th percentile dropped further to
~56ms.

Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Reviewed-on: http://gerrit.cloudera.org:8080/3184
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans
Reviewed-by: Adar Dembo <ad...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/8094b731
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/8094b731
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/8094b731

Branch: refs/heads/master
Commit: 8094b73147e0238cdfd96a141554b5344c801cdb
Parents: 298806f
Author: Todd Lipcon <to...@cloudera.com>
Authored: Mon May 23 21:43:31 2016 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue May 24 20:17:26 2016 +0000

----------------------------------------------------------------------
 .../src/main/java/org/kududb/client/AsyncKuduClient.java        | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/8094b731/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java b/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
index f67f8a3..d237e3b 100644
--- a/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
+++ b/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
@@ -921,8 +921,9 @@ public class AsyncKuduClient implements AutoCloseable {
           new Exception("Exception created to collect stack trace"));
       attemptCount = 1;
     }
-    // TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50
-    long sleepTime = (attemptCount * SLEEP_TIME) + sleepRandomizer.nextInt(50);
+    // Randomized exponential backoff, truncated at 4096ms.
+    long sleepTime = (long)(Math.pow(2.0, Math.min(attemptCount, 12))
+        * sleepRandomizer.nextDouble());
     if (LOG.isDebugEnabled()) {
       LOG.debug("Going to sleep for " + sleepTime + " at retry " + rpc.attempt);
     }