You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2019/04/30 22:26:38 UTC
[kudu-CR] Deflake ClientTest.TestServerTooBusyRetry
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13200
Change subject: Deflake ClientTest.TestServerTooBusyRetry
......................................................................
Deflake ClientTest.TestServerTooBusyRetry
ClientTest.TestServerTooBusyRetry is a mess of a test. In TSAN mode,
there are less rows inserted, so scans require less round trips to
complete, but at the same time threads start slower, so the number of
scans in-flight at once will tend to be lower. This causes the test to
occasionally fail to cause a service queue overflow, as it is intended
to do. Eventually, the test fails because TSAN has an upper bound on the
number of threads that can be created in the lifetime of a single TSAN
process, and the test slowly creates scan threads.
This patch attempts to address the problem by raising the scan batch
latency in TSAN mode. With this patch, I saw 0 failures in 1000 runs.
Without it, I got tired of waiting for 850/1000 to finish after 15
minutes.
This is a quick fix. In the future someone should consider a more
serious rewrite of this test.
Change-Id: Id4d2ee077e9d107fb475c399af5690084bdeef49
---
M src/kudu/client/client-test.cc
1 file changed, 4 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/00/13200/1
--
To view, visit http://gerrit.cloudera.org:8080/13200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id4d2ee077e9d107fb475c399af5690084bdeef49
Gerrit-Change-Number: 13200
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
[kudu-CR] Deflake ClientTest.TestServerTooBusyRetry
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13200 )
Change subject: Deflake ClientTest.TestServerTooBusyRetry
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13200/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13200/1//COMMIT_MSG@23
PS1, Line 23: This is a quick fix.
An alternative may be to cap the number of outstanding threads. Or to have a fixed number of threads loop scanning vs. doing just one scan per thread. But this fix is fine to me.
--
To view, visit http://gerrit.cloudera.org:8080/13200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4d2ee077e9d107fb475c399af5690084bdeef49
Gerrit-Change-Number: 13200
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 30 Apr 2019 23:07:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] Deflake ClientTest.TestServerTooBusyRetry
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13200 )
Change subject: Deflake ClientTest.TestServerTooBusyRetry
......................................................................
Deflake ClientTest.TestServerTooBusyRetry
ClientTest.TestServerTooBusyRetry is a mess of a test. In TSAN mode,
there are less rows inserted, so scans require less round trips to
complete, but at the same time threads start slower, so the number of
scans in-flight at once will tend to be lower. This causes the test to
occasionally fail to cause a service queue overflow, as it is intended
to do. Eventually, the test fails because TSAN has an upper bound on the
number of threads that can be created in the lifetime of a single TSAN
process, and the test slowly creates scan threads.
This patch attempts to address the problem by raising the scan batch
latency in TSAN mode. With this patch, I saw 0 failures in 1000 runs.
Without it, I got tired of waiting for 850/1000 to finish after 15
minutes.
This is a quick fix. In the future someone should consider a more
serious rewrite of this test.
Change-Id: Id4d2ee077e9d107fb475c399af5690084bdeef49
Reviewed-on: http://gerrit.cloudera.org:8080/13200
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
M src/kudu/client/client-test.cc
1 file changed, 4 insertions(+), 0 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/13200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id4d2ee077e9d107fb475c399af5690084bdeef49
Gerrit-Change-Number: 13200
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>