You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2018/04/30 15:23:33 UTC
[kudu-CR] [tests] fixed flake in consensus peer health status
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10237
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
[tests] fixed flake in consensus_peer_health_status
Fixed flake in the TestPeerHealthStatusTransitions scenario of the
ConsensusPeerHealthStatusITest test. Prior to the fix, the flakiness
happened when the target tablet server was shutdown during an on-going
tablet copy. In that situation, the source tablet server had
corresponding WAL segments anchored, so they could not be GCed.
As a result, the tablet replica would not get FAILED_UNRECOVERABLE
health status assigned.
Test results using dist-test, before and after the fix (ASAN),
before:
http://dist-test.cloudera.org/job?job_id=aserbin.1525098763.54471
after:
http://dist-test.cloudera.org/job?job_id=aserbin.1525099526.63998
Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
---
M src/kudu/integration-tests/consensus_peer_health_status-itest.cc
1 file changed, 1 insertion(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/10237/1
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [tests] fixed flake in consensus peer health status
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10237
to look at the new patch set (#2).
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
[tests] fixed flake in consensus_peer_health_status
Fixed flake in the TestPeerHealthStatusTransitions scenario of the
ConsensusPeerHealthStatusITest test. Prior to the fix, the flakiness
happened when the target tablet server was shutdown during an on-going
tablet copy, where the tablet copy was initiated by AddServer() call
while preparing the mini-cluster for the peer health sequence of
"HEALTHY -> UNKNOWN -> FAILED -> FAILED_UNRECOVERABLE".
In that situation, the source tablet server had corresponding WAL
segments anchored, so they could not be GCed. As a result, the tablet
replica would not get FAILED_UNRECOVERABLE health status in 30 seconds
because --tablet_copy_idle_timeout_sec is set to 600 seconds by default.
Test results using dist-test, before and after the fix (ASAN),
before:
http://dist-test.cloudera.org/job?job_id=aserbin.1525111507.120645
after:
http://dist-test.cloudera.org/job?job_id=aserbin.1525099526.63998
Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
---
M src/kudu/integration-tests/consensus_peer_health_status-itest.cc
1 file changed, 6 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/10237/2
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [tests] fixed flake in consensus peer health status
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10237 )
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10237/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10237/1//COMMIT_MSG@11
PS1, Line 11: happened when the target tablet server was shutdown during an on-going
: tablet copy. In that situation, the source tablet server had
: c
> curious why you took this approach instead of just waiting for the tablet c
Ah, it seems I omitted some useful information on where the flake happened, I'll add that.
The flake happened in the last line of the test scenario, and , as I understand, the tablet copy that hold anchor as triggered by AddServer() at line 197.
Yes, I tried ASSERT_OK(WaitUntilTabletRunning(follower_ts, tablet_id_, kTimeout)) instead of NO_FATALS(wait_for_health_state(leader_ts, follower_uuid, HEALTHY)) at line 198 and it also worked. I thought maybe it's worth keeping this interesting situation with hung tablet copy (even if it's rare).
From the other side, if we want to be explicit and just start from scratch, then WaitUntilTabletRunning() after AddServer() is the best approach, I think. All right, I'll change to that one.
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 30 Apr 2018 18:09:01 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tests] fixed flake in consensus peer health status
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10237 )
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 30 Apr 2018 22:50:17 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] fixed flake in consensus peer health status
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10237 )
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10237/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10237/1//COMMIT_MSG@11
PS1, Line 11: happened when the target tablet server was shutdown during an on-going
: tablet copy. In that situation, the source tablet server had
: c
curious why you took this approach instead of just waiting for the tablet copy to finish? The tablet copy here is caused by the prior part of the test, right?
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 30 Apr 2018 16:41:02 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tests] fixed flake in consensus peer health status
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10237 )
Change subject: [tests] fixed flake in consensus_peer_health_status
......................................................................
[tests] fixed flake in consensus_peer_health_status
Fixed flake in the TestPeerHealthStatusTransitions scenario of the
ConsensusPeerHealthStatusITest test. Prior to the fix, the flakiness
happened when the target tablet server was shutdown during an on-going
tablet copy, where the tablet copy was initiated by AddServer() call
while preparing the mini-cluster for the peer health sequence of
"HEALTHY -> UNKNOWN -> FAILED -> FAILED_UNRECOVERABLE".
In that situation, the source tablet server had corresponding WAL
segments anchored, so they could not be GCed. As a result, the tablet
replica would not get FAILED_UNRECOVERABLE health status in 30 seconds
because --tablet_copy_idle_timeout_sec is set to 600 seconds by default.
Test results using dist-test, before and after the fix (ASAN),
before:
http://dist-test.cloudera.org/job?job_id=aserbin.1525111507.120645
after:
http://dist-test.cloudera.org/job?job_id=aserbin.1525099526.63998
Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Reviewed-on: http://gerrit.cloudera.org:8080/10237
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mp...@apache.org>
---
M src/kudu/integration-tests/consensus_peer_health_status-itest.cc
1 file changed, 6 insertions(+), 2 deletions(-)
Approvals:
Kudu Jenkins: Verified
Mike Percy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/10237
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8eb640604e98361029aa3342ffa3050e922b6629
Gerrit-Change-Number: 10237
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>