[2/5] kudu git commit: rpc-test: fix flakiness in tests that expect timeouts

[mp...@apache.org]

rpc-test: fix flakiness in tests that expect timeouts

The DoTestExpectTimeout() utility function would set a timeout to some
number of milliseconds N, and then ask the server to sleep for (N + 50)
milliseconds, expecting a timeout. It would then assert that the timeout
was returned after some amount of time between N and N+50, but no more
than N+50ms.

This would be flaky under concurrent load (eg stress threads) because
the sleep(50ms) might sometimes actually sleep for an extra 50-100ms.

This just changes the test to ask the server to sleep for n+500ms,
giving it a lot more budget for sloppiness.

I looped TestCallTimeout/0 with 4 stress threads 1000 times in TSAN.
Before[1] it failed 4/1000. After[2] it didn't fail.

[1] http://dist-test.cloudera.org//job?job_id=todd.1480912345.2054
[2] http://dist-test.cloudera.org//job?job_id=todd.1480912461.12007

Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c
Reviewed-on: http://gerrit.cloudera.org:8080/5356
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/916c21be
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/916c21be
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/916c21be

Branch: refs/heads/master
Commit: 916c21be1846effb5325614f278eb4ace23f7f37
Parents: 0a89833
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Dec 5 12:34:18 2016 +0800
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Mon Dec 5 17:32:38 2016 +0000

----------------------------------------------------------------------
 src/kudu/rpc/rpc-test-base.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/916c21be/src/kudu/rpc/rpc-test-base.h
----------------------------------------------------------------------
diff --git a/src/kudu/rpc/rpc-test-base.h b/src/kudu/rpc/rpc-test-base.h
index 133cc14..c6c87a8 100644
--- a/src/kudu/rpc/rpc-test-base.h
+++ b/src/kudu/rpc/rpc-test-base.h
@@ -483,8 +483,8 @@ class RpcTestBase : public KuduTest {
   void DoTestExpectTimeout(const Proxy &p, const MonoDelta &timeout) {
     SleepRequestPB req;
     SleepResponsePB resp;
-    // Sleep for 50ms longer than the call timeout.
-    int sleep_micros = timeout.ToMicroseconds() + 50000;
+    // Sleep for 500ms longer than the call timeout.
+    int sleep_micros = timeout.ToMicroseconds() + 500 * 1000;
     req.set_sleep_micros(sleep_micros);
 
     RpcController c;