You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2017/10/17 17:01:14 UTC
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8298
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
[itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Fixed flakiness in the TombstonedVoteAfterFailedTabletCopy scenario
of the raft_consensus_election-itest. A race between adding a new
replica and DeleteTablet was possible if adding a new server before
the catalog manager acknowledges eviction of the former replica falling
behind the leader's log. In short, delayed DeleteTablet request might
be sent/processed _after_ a new server has been added.
The race was easily reproducible by running the test scenario with
high number of stress-cpu-threads:
./bin/raft_consensus_election-itest \
--stress_cpu_threads=128 \
--gtest_filter=\
'RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy'
Prior to this change, the test failed almost 100% if running as
described above. After the fix, no failures observed in about 100 runs.
Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
---
M src/kudu/integration-tests/raft_consensus_election-itest.cc
1 file changed, 20 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/8298/1
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
Patch Set 1: Verified+1
Unrelated flake in ClientStressTest.TestUniqueClientIds
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 17 Oct 2017 17:54:35 +0000
Gerrit-HasComments: No
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8298
to look at the new patch set (#2).
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
[itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Fixed flakiness in the TombstonedVoteAfterFailedTabletCopy scenario
of the raft_consensus_election-itest. A race between adding a new
replica and DeleteTablet was possible if adding a new server before
the catalog manager acknowledges eviction of the former replica and
sends DeleteTablet request. In other words, prior to this fix,
a delayed DeleteTablet request might be processed _after_ a new
server has been added.
The race was easily reproducible by running the test scenario with
high number of stress-cpu-threads:
./bin/raft_consensus_election-itest \
--stress_cpu_threads=128 \
--gtest_filter=\
'RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy'
Prior to this change, the test failed almost 100% if running as
described above. After the fix, no failures observed in about 100 runs.
Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
---
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/integration-tests/raft_consensus_election-itest.cc
2 files changed, 17 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/8298/2
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 18 Oct 2017 21:31:22 +0000
Gerrit-HasComments: No
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
[itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Fixed flakiness in the TombstonedVoteAfterFailedTabletCopy scenario
of the raft_consensus_election-itest. A race between adding a new
replica and DeleteTablet was possible if adding a new server before
the catalog manager acknowledges eviction of the former replica and
sends DeleteTablet request. In other words, prior to this fix,
a delayed DeleteTablet request might be processed _after_ a new
server has been added.
The race was easily reproducible by running the test scenario with
high number of stress-cpu-threads:
./bin/raft_consensus_election-itest \
--stress_cpu_threads=128 \
--gtest_filter=\
'RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy'
Prior to this change, the test failed almost 100% if running as
described above. After the fix, no failures observed in about 100 runs.
Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Reviewed-on: http://gerrit.cloudera.org:8080/8298
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/integration-tests/raft_consensus_election-itest.cc
2 files changed, 17 insertions(+), 7 deletions(-)
Approvals:
Todd Lipcon: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
Removed reviewer Kudu Jenkins with the following votes:
* Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8298/1/src/kudu/integration-tests/raft_consensus_election-itest.cc
File src/kudu/integration-tests/raft_consensus_election-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8298/1/src/kudu/integration-tests/raft_consensus_election-itest.cc@510
PS1, Line 510: ASSERT_OK(itest::WaitForReplicasReportedToMaster(
would it be more direct to just ASSERT_EVENTUALLY waiting for the data state on that removed server to go to TOMBSTONED state? that matches more closely what the comment above is saying. Waiting for it to be reported to the catalog manager still doesn't necessarily mean that the DeleteTablet call has gotten sent out and processed by the replica.
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 18 Oct 2017 19:29:13 +0000
Gerrit-HasComments: Yes
[kudu-CR] [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8298 )
Change subject: [itests] fix flake in TombstonedVoteAfterFailedTabletCopy
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8298/1/src/kudu/integration-tests/raft_consensus_election-itest.cc
File src/kudu/integration-tests/raft_consensus_election-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8298/1/src/kudu/integration-tests/raft_consensus_election-itest.cc@510
PS1, Line 510: ASSERT_OK(itest::WaitForReplicasReportedToMaster(
> would it be more direct to just ASSERT_EVENTUALLY waiting for the data stat
Good idea -- that would be 100% guarantee of proper event sequencing and race-free approach.
Yep, I understand the proposed solution was not 100% reliable; it seems I just didn't connects all dots when looking at the failure.
--
To view, visit http://gerrit.cloudera.org:8080/8298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifaba16d1acfee6ff01e475525482f21ff64133f4
Gerrit-Change-Number: 8298
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 18 Oct 2017 21:29:49 +0000
Gerrit-HasComments: Yes