You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2025/04/18 20:54:06 UTC
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22797
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
[rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Before this patch, the TestRpc.TimedOutOnResponseMetricServiceQueue
scenario would fail from time to time (about once in every 100 runs)
when running on a VM in AWS cloud with error message like below:
src/kudu/rpc/rpc-test.cc:1499: Failure
Value of: s.IsTimedOut()
Actual: false
Expected: true
Remote error: Got some error
I haven't seen it if running on nodes backed by a dedicated hardware,
and I guess that's due to scheduler anomalies induced by shared virtual
environment. However, I didn't look deeper and just increased the sleep
time 10x and added 2x margin for RPC timeout. With this patch, there
hasn't been a single failure in more than 10K runs of the test scenario.
The scenario still runs quite fast on modern hardware: less than 500ms.
Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Reviewed-on: http://gerrit.cloudera.org:8080/22747
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
(cherry picked from commit e65914e386339a11e61d5a8cb24d66bf606fd41a)
---
M src/kudu/rpc/rpc-test.cc
1 file changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/22797/1
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22797 )
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/1293/
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 18 Apr 2025 20:54:13 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/22797 )
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 21 Apr 2025 21:16:44 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22797 )
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/1293/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 18 Apr 2025 21:50:12 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/22797 )
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
Patch Set 1: Verified+1
unrelated java test failure (RELEASE): org.apache.kudu.spark.kudu.DefaultSourceTest
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 18 Apr 2025 23:38:53 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22797 )
Change subject: [rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
......................................................................
[rpc-test] fix flakiness in TimedOutOnResponseMetricServiceQueue
Before this patch, the TestRpc.TimedOutOnResponseMetricServiceQueue
scenario would fail from time to time (about once in every 100 runs)
when running on a VM in AWS cloud with error message like below:
src/kudu/rpc/rpc-test.cc:1499: Failure
Value of: s.IsTimedOut()
Actual: false
Expected: true
Remote error: Got some error
I haven't seen it if running on nodes backed by a dedicated hardware,
and I guess that's due to scheduler anomalies induced by shared virtual
environment. However, I didn't look deeper and just increased the sleep
time 10x and added 2x margin for RPC timeout. With this patch, there
hasn't been a single failure in more than 10K runs of the test scenario.
The scenario still runs quite fast on modern hardware: less than 500ms.
Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Reviewed-on: http://gerrit.cloudera.org:8080/22747
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
(cherry picked from commit e65914e386339a11e61d5a8cb24d66bf606fd41a)
Reviewed-on: http://gerrit.cloudera.org:8080/22797
---
M src/kudu/rpc/rpc-test.cc
1 file changed, 4 insertions(+), 2 deletions(-)
Approvals:
Alexey Serbin: Verified
Abhishek Chennaka: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/22797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I70008836a38def70e097bc4547ac2a66e7203e35
Gerrit-Change-Number: 22797
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)