You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2018/03/26 23:11:58 UTC
[kudu-CR](branch-1.7.x) [linked list-test] fix flake with the 3-4-3 scheme
Hello Mike Percy,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/9811
to review the following change.
Change subject: [linked_list-test] fix flake with the 3-4-3 scheme
......................................................................
[linked_list-test] fix flake with the 3-4-3 scheme
The LinkedListTest.TestLoadWhileOneServerDownAndVerify scenario became
flaky when running with --seconds_to_run set to about 800 and more
once the 3-4-3 replica management scheme became the default one.
Those cases (--seconds_to_run=X, X >= 800) are special because the
stopped replica falls behind WAL segment GC threshold when running with
the linked list input data, so the system automatically replaces
the failed replica. In case of the 3-4-3 scheme, the newly added
replica is added as a non-voter. The WaitForServersToAgree() looks
only at the OpId indices, not distinguishing between voter and
non-voter replicas. However, the verification phase of the scenario
assumes the only replica left alive is a voter replica.
Prior to this fix, the scenario didn't account for the case when the
replica at the restarted tablet server was still a non-voter, and in
most cases the rest 2 out of 3 tservers were shutdown before the newly
added replica was promoted. As a result, the latter replica was left
non-voter and the written data could not be read back from it.
This patch adds a step to verify that all 3 replicas are registered
as voters with the master(s) before shutting down the tservers hosting
the source 2 replicas. The scenario is now stable when running with
--seconds_to_run=900.
Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Reviewed-on: http://gerrit.cloudera.org:8080/9795
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Mike Percy <mp...@apache.org>
(cherry picked from commit a781710264da43a1938aff2fffd112342efd41c9)
---
M src/kudu/integration-tests/linked_list-test.cc
1 file changed, 31 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/11/9811/1
--
To view, visit http://gerrit.cloudera.org:8080/9811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.7.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Gerrit-Change-Number: 9811
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR](branch-1.7.x) [linked list-test] fix flake with the 3-4-3 scheme
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9811 )
Change subject: [linked_list-test] fix flake with the 3-4-3 scheme
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/9811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.7.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Gerrit-Change-Number: 9811
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Tue, 27 Mar 2018 00:28:52 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.7.x) [linked list-test] fix flake with the 3-4-3 scheme
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/9811 )
Change subject: [linked_list-test] fix flake with the 3-4-3 scheme
......................................................................
Patch Set 1: Verified+1
The same unrelated flake with libtool:
libtool: Version mismatch error. This is libtool 2.4.2 Debian-2.4.2-1.7ubuntu1, but the definition of this LT_INIT comes from libtool 2.4.6. You should recreate aclocal.m4 with macros from libtool 2.4.2 Debian-2.4.2-1.7ubuntu1
--
To view, visit http://gerrit.cloudera.org:8080/9811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.7.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Gerrit-Change-Number: 9811
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Tue, 27 Mar 2018 00:28:11 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.7.x) [linked list-test] fix flake with the 3-4-3 scheme
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/9811 )
Change subject: [linked_list-test] fix flake with the 3-4-3 scheme
......................................................................
Removed reviewer Kudu Jenkins with the following votes:
* Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/9811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.7.x
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Gerrit-Change-Number: 9811
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR](branch-1.7.x) [linked list-test] fix flake with the 3-4-3 scheme
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9811 )
Change subject: [linked_list-test] fix flake with the 3-4-3 scheme
......................................................................
[linked_list-test] fix flake with the 3-4-3 scheme
The LinkedListTest.TestLoadWhileOneServerDownAndVerify scenario became
flaky when running with --seconds_to_run set to about 800 and more
once the 3-4-3 replica management scheme became the default one.
Those cases (--seconds_to_run=X, X >= 800) are special because the
stopped replica falls behind WAL segment GC threshold when running with
the linked list input data, so the system automatically replaces
the failed replica. In case of the 3-4-3 scheme, the newly added
replica is added as a non-voter. The WaitForServersToAgree() looks
only at the OpId indices, not distinguishing between voter and
non-voter replicas. However, the verification phase of the scenario
assumes the only replica left alive is a voter replica.
Prior to this fix, the scenario didn't account for the case when the
replica at the restarted tablet server was still a non-voter, and in
most cases the rest 2 out of 3 tservers were shutdown before the newly
added replica was promoted. As a result, the latter replica was left
non-voter and the written data could not be read back from it.
This patch adds a step to verify that all 3 replicas are registered
as voters with the master(s) before shutting down the tservers hosting
the source 2 replicas. The scenario is now stable when running with
--seconds_to_run=900.
Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Reviewed-on: http://gerrit.cloudera.org:8080/9795
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Mike Percy <mp...@apache.org>
(cherry picked from commit a781710264da43a1938aff2fffd112342efd41c9)
Reviewed-on: http://gerrit.cloudera.org:8080/9811
---
M src/kudu/integration-tests/linked_list-test.cc
1 file changed, 31 insertions(+), 14 deletions(-)
Approvals:
Alexey Serbin: Verified
Mike Percy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/9811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.7.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I132206371e2935f1e0f39e9eacad866fde22c5b8
Gerrit-Change-Number: 9811
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>