You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2020/01/28 03:52:39 UTC
[kudu-CR] KUDU-3046: dfelake TabletServerQuiescingITest
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15113
Change subject: KUDU-3046: dfelake TabletServerQuiescingITest
......................................................................
KUDU-3046: dfelake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting exact the number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist.
- Addressed this by stepping down on all tablet servers just to be
sure.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 26 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/15113/1
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15113
to look at the new patch set (#4).
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting the exact number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist even after stepping down.
- Addressed this by stepping down on all tablet servers just to be
sure, and retrying if necessary via ASSERT_EVENTUALLY.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 45 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/15113/4
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15113
to look at the new patch set (#2).
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting exact the number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist.
- Addressed this by stepping down on all tablet servers just to be
sure.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 26 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/15113/2
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting the exact number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist even after stepping down.
- Addressed this by stepping down on all tablet servers just to be
sure, and retrying if necessary via ASSERT_EVENTUALLY.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Reviewed-on: http://gerrit.cloudera.org:8080/15113
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 43 insertions(+), 11 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15113/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15113/3//COMMIT_MSG@10
PS3, Line 10: - Slowness in TSAN mode along with a low Raft timeout meant workloads
: would fail to even create tablets.
: - Addressed this by increasing the heartbeat interval in TSAN mode.
> This happens to us quite often. How do you feel about adding a DCHECK or so
I think it'd be pretty difficult to enforce this is doing the right thing. Sometimes the super low timeout is exactly what we want, eg in stress tests.
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc
File src/kudu/integration-tests/tablet_server_quiescing-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@a197
PS3, Line 197:
> Yup, we have some test which run with ranges 25ms -- 200ms hearbeat interva
It wasn't really necessary once we began more aggressively transferring leadership. Though I suppose it doesn't hurt to increase the frequency of Raft communication since it still means waiting 100ms instead of 500ms.
Done
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@699
PS3, Line 699: // step it down, but it's hard to guarantee that the found leader is of the
> Indeed, what if the tablet servers were in election cycle already when a re
I think the 2x wait time might still be insufficient because there still might be multiple leaders of different terms when we go out to find a leader.
I opted to go with Adar's approach rather than sleeping. I also wrapped this in an ASSERT_EVENTUALLY for good measure.
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 28 Jan 2020 21:07:03 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc
File src/kudu/integration-tests/tablet_server_quiescing-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@699
PS3, Line 699: // latest term.
> I think the 2x wait time might still be insufficient because there still mi
+1
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 28 Jan 2020 22:44:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc
File src/kudu/integration-tests/tablet_server_quiescing-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@a197
PS3, Line 197:
> Why don't want to lower the heartbeat interval in this test any more?
Yup, we have some test which run with ranges 25ms -- 200ms hearbeat interval. It seems 25ms might be a good option, at least 50ms seems be viable for non-sanitizer builds.
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@699
PS3, Line 699: ignore_result(LeaderStepDown(ts_and_details.second, tablet_id, kLeaderTimeout));
> Should we ASSERT that we succeeded at least once somewhere?
Indeed, what if the tablet servers were in election cycle already when a request to quiesce arrived? They could vote, and then receive the quiesce request. The step-down request might arrive to the leader before it got 'yes' votes from the followers, right?
Would it help to wait for about 2x of time necessary to elect a leader after sending last quiesce request and before first LeaderStepDown() call?
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 28 Jan 2020 04:55:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15113/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15113/3//COMMIT_MSG@10
PS3, Line 10: - Slowness in TSAN mode along with a low Raft timeout meant workloads
: would fail to even create tablets.
: - Addressed this by increasing the heartbeat interval in TSAN mode.
This happens to us quite often. How do you feel about adding a DCHECK or something equivalent to enforce that, in TSAN, the Raft heartbeat interval isn't below some threshold?
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc
File src/kudu/integration-tests/tablet_server_quiescing-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@a197
PS3, Line 197:
Why don't want to lower the heartbeat interval in this test any more?
http://gerrit.cloudera.org:8080/#/c/15113/3/src/kudu/integration-tests/tablet_server_quiescing-itest.cc@699
PS3, Line 699: ignore_result(LeaderStepDown(ts_and_details.second, tablet_id, kLeaderTimeout));
Should we ASSERT that we succeeded at least once somewhere?
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 28 Jan 2020 04:35:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15113
to look at the new patch set (#3).
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting the exact number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist.
- Addressed this by stepping down on all tablet servers just to be
sure.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 26 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/15113/3
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 28 Jan 2020 21:18:05 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 28 Jan 2020 22:02:30 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3046: deflake TabletServerQuiescingITest
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15113
to look at the new patch set (#5).
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting the exact number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist even after stepping down.
- Addressed this by stepping down on all tablet servers just to be
sure, and retrying if necessary via ASSERT_EVENTUALLY.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 43 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/15113/5
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)