You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2018/05/17 20:05:58 UTC
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10443
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet immediately after it is
created flaky, so we simply retry.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/tserver/tablet_server-test.cc
1 file changed, 10 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/1
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc
File src/kudu/tserver/tablet_server-test-base.cc:
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc@346
PS4, Line 346: void TabletServerTestBase::TombstoneTablet() {
> Aye, gotcha. There's currently nothing in TSTabletManager that exposes tran
Yeah I think that would be acceptable.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 22 May 2018 20:55:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc
File src/kudu/tserver/tablet_server-test-base.cc:
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc@346
PS4, Line 346: void TabletServerTestBase::TombstoneTablet() {
This isn't what I had in mind; sorry for not explaining well.
I was thinking that since the test fixture is responsible for creating the tablet, it should also _wait for the tablet transition marker to be removed_ before handing control over to the test. Then the particular test that's flaky wouldn't need any changes (though the other one may still need an ASSERT_EVENTUALLY.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 22 May 2018 20:05:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has removed a vote on this change.
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc
File src/kudu/tserver/tablet_server-test-base.cc:
http://gerrit.cloudera.org:8080/#/c/10443/4/src/kudu/tserver/tablet_server-test-base.cc@346
PS4, Line 346: void TabletServerTestBase::TombstoneTablet() {
> This isn't what I had in mind; sorry for not explaining well.
Aye, gotcha. There's currently nothing in TSTabletManager that exposes transitions in progress (besides getting rejected while calling some methods that cause state transitions). Do you think we should add some 'WaitForNoTransitionsForTests' method?
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 22 May 2018 20:33:03 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 6: Verified+1
Looks like some Java flake.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 25 May 2018 20:43:42 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10443
to look at the new patch set (#5).
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet without rety immediately
after it is created flaky. To fix this for tests using an
in-process tablet server, I added a method to the TSTabletManager
that waits for there to be no tablet transitions in progress.
This method is intended for use only in tests, and is used by the
tablet server test fixture to ensure its single tablet has transitioned
out of the creating state before the actual test runs.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue. I changed it to retry the delete, as
this tests uses an external minicluster.
I ran the affected tests 1000 times with 8 stress threads in
debug mode in dist-test. All passed.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
5 files changed, 43 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/5
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10443
to look at the new patch set (#3).
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet immediately after it is
created flaky. To fix this, I added a method to the test
fixture which will simply retry the delete.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue. I changed it to retry the delete.
I ran the affected tests 1000 times with 8 stress threads in
debug mode in dist-test. All passed.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test-base.h
M src/kudu/tserver/tablet_server-test.cc
4 files changed, 27 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/3
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc@541
PS1, Line 541: // Even though the test setup guarantees the tablet is running, the
: // transition marker may not have been deleted yet, so we retry
: // tombstoning the tablet until it is.
> I was wrong about that as initially I thought the race was more serious.
Okay. What do you think of my first suggestion: moving this into the test fixture?
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 21 May 2018 17:11:49 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10443
to look at the new patch set (#6).
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet without retry immediately
after it is created flaky. To fix this for tests using an
in-process tablet server, I added a method to the TSTabletManager
that waits for there to be no tablet transitions in progress.
This method is intended for use only in tests, and is used by the
tablet server test fixture to ensure its single tablet has transitioned
out of the creating state before the actual test runs.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue. I changed it to retry the delete, as
this tests uses an external minicluster.
I ran the affected tests 1000 times with 8 stress threads in
debug mode in dist-test. All passed.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
5 files changed, 42 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/6
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10443/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10443/5//COMMIT_MSG@11
PS5, Line 11: ely
> retry
Done
http://gerrit.cloudera.org:8080/#/c/10443/5/src/kudu/tserver/tablet_server-test-base.cc
File src/kudu/tserver/tablet_server-test-base.cc:
http://gerrit.cloudera.org:8080/#/c/10443/5/src/kudu/tserver/tablet_server-test-base.cc@138
PS5, Line 138: replica->consensus()->WaitUntilLeaderFor
> Could use tablet_manager here instead.
(facepalm) Made the variable to use it in both places and forgot to change the first.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 25 May 2018 19:11:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10443
to look at the new patch set (#4).
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet immediately after it is
created flaky. To fix this, I added a method to the test
fixture which will simply retry the delete.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue. I changed it to retry the delete.
I ran the affected tests 1000 times with 8 stress threads in
debug mode in dist-test. All passed.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test-base.h
M src/kudu/tserver/tablet_server-test.cc
4 files changed, 28 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/4
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 1:
Would like to run this with dist-test to verify flakiness is gone.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 17 May 2018 20:07:08 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc@541
PS1, Line 541: mini_server_->Shutdown();
: ASSERT_OK(mini_server_->Restart());
: ASSERT_OK(mini_server_->WaitStarted())
> Okay. What do you think of my first suggestion: moving this into the test f
Yup, good idea. It applies here and to another test in this file. The tablet copy itest that also had this problem doesn't use the same fixture, and it running vs an EMC, so I think retrying in the test is appropriate there.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 22 May 2018 18:05:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 1:
Saw the same problem on another precommit: http://jenkins.kudu.apache.org/job/kudu-gerrit/13536/BUILD_TYPE=ASAN/testReport/junit/(root)/TabletCopyClientSessionITest/TestCopyFromCrashedSource/
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 18 May 2018 23:29:57 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc@541
PS1, Line 541: // Even though the test setup guarantees the tablet is running, the
: // transition marker may not have been deleted yet, so we retry
: // tombstoning the tablet until it is.
> Since the creation of kTabletId is the responsibility of the test fixture,
I was wrong about that as initially I thought the race was more serious.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 17 May 2018 20:49:17 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/10443/1/src/kudu/tserver/tablet_server-test.cc@541
PS1, Line 541: // Even though the test setup guarantees the tablet is running, the
: // transition marker may not have been deleted yet, so we retry
: // tombstoning the tablet until it is.
Since the creation of kTabletId is the responsibility of the test fixture, shouldn't it also be responsible for this too?
On Slack you mentioned that other tests may be vulnerable to this. What needs to happen for a comprehensive fix?
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 17 May 2018 20:28:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 25 May 2018 19:16:03 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet without retry immediately
after it is created flaky. To fix this for tests using an
in-process tablet server, I added a method to the TSTabletManager
that waits for there to be no tablet transitions in progress.
This method is intended for use only in tests, and is used by the
tablet server test fixture to ensure its single tablet has transitioned
out of the creating state before the actual test runs.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue. I changed it to retry the delete, as
this tests uses an external minicluster.
I ran the affected tests 1000 times with 8 stress threads in
debug mode in dist-test. All passed.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Reviewed-on: http://gerrit.cloudera.org:8080/10443
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Will Berkeley <wd...@gmail.com>
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
5 files changed, 42 insertions(+), 9 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Will Berkeley: Verified
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 7
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10443 )
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10443/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10443/5//COMMIT_MSG@11
PS5, Line 11: rety
retry
http://gerrit.cloudera.org:8080/#/c/10443/5/src/kudu/tserver/tablet_server-test-base.cc
File src/kudu/tserver/tablet_server-test-base.cc:
http://gerrit.cloudera.org:8080/#/c/10443/5/src/kudu/tserver/tablet_server-test-base.cc@138
PS5, Line 138: mini_server_->server()->tablet_manager()
Could use tablet_manager here instead.
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 25 May 2018 17:16:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10443
to look at the new patch set (#2).
Change subject: KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
......................................................................
KUDU-2444 TabletServerTest.TestTombstonedTabletOnWebUI is flaky
It's possible for a tablet to be RUNNING with working consensus
but still have its "creating" transition live in the tablet
manager. This makes deleting a tablet immediately after it is
created flaky, so we simply retry.
In addition to the original report from tablet_server-test.cc,
I also saw a test in tablet_copy_client_session-itest.cc fail
because of the same issue.
Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
---
M src/kudu/integration-tests/tablet_copy_client_session-itest.cc
M src/kudu/tserver/tablet_server-test.cc
2 files changed, 16 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/10443/2
--
To view, visit http://gerrit.cloudera.org:8080/10443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib8dc7acc4c39f9d9b067a52df4bd6732f545a8de
Gerrit-Change-Number: 10443
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>