You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2024/10/15 21:11:18 UTC
[kudu-CR] [dynamic multi master-test] reduce amount of generated data
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21935
Change subject: [dynamic_multi_master-test] reduce amount of generated data
......................................................................
[dynamic_multi_master-test] reduce amount of generated data
When I ran all C++ tests via ctest on one node having about 25GiBytes of
available disk space, it unexpectedly ran out of space. The culprit was
dynamic_multi_master-test: the scenarios from the suite keep their test
workloads running while waiting for events that might take a long time
to happen, especially on slow or very busy machines.
This patch updates the configuration of the TestWorkload objects to use
a single writer thread (the default is four). In addition,
a bit of extra delay is now added between writing batches of test rows
in the TestBasicAddition scenario.
Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
---
M src/kudu/master/dynamic_multi_master-test.cc
1 file changed, 10 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/21935/1
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has removed a vote on this change.
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [dynamic multi master-test] reduce amount of generated data
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce amount of generated data
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/591/
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2024 21:11:27 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Abhishek Chennaka, KeDeng,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21935
to look at the new patch set (#2).
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
[dynamic_multi_master-test] reduce the amount of generated data
When I ran all C++ tests via ctest on one node having about 25GiBytes of
available disk space, it unexpectedly ran out of space. The culprit was
dynamic_multi_master-test: the scenarios from the suite keep their test
workloads running while waiting for events that might take a long time
to happen, especially on slow or very busy machines.
This patch updates the configuration of the TestWorkload objects to use
a single writer thread (the default is four) in the TestBasicAddition
scenario of the ParameterizedAutoAddMasterTest test. In addition,
a bit of extra delay is now added between writing batches of test rows.
Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
---
M src/kudu/master/dynamic_multi_master-test.cc
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/21935/2
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/592/
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2024 21:29:22 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21935/2/src/kudu/master/dynamic_multi_master-test.cc
File src/kudu/master/dynamic_multi_master-test.cc:
http://gerrit.cloudera.org:8080/#/c/21935/2/src/kudu/master/dynamic_multi_master-test.cc@1833
PS2, Line 1833: quite a long time
Do we want to skip slow tests in that case using SKIP_IF_SLOW_NOT_ALLOWED()?
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 15:51:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 1:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/591/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 15 Oct 2024 21:29:13 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 08:58:03 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 13:29:47 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21935/2/src/kudu/master/dynamic_multi_master-test.cc
File src/kudu/master/dynamic_multi_master-test.cc:
http://gerrit.cloudera.org:8080/#/c/21935/2/src/kudu/master/dynamic_multi_master-test.cc@1833
PS2, Line 1833: quite a long time
> Do we want to skip slow tests in that case using SKIP_IF_SLOW_NOT_ALLOWED()
That's a good point.
Yep, I can take a closer look at this and other scenarios, adding SKIP_IF_SLOW_NOT_ALLOWED() where needed (e.g., where the runtime is 3 seconds and more). I will post it in a separate patch.
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 16:58:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
[dynamic_multi_master-test] reduce the amount of generated data
When I ran all C++ tests via ctest on one node having about 25GiBytes of
available disk space, it unexpectedly ran out of space. The culprit was
dynamic_multi_master-test: the scenarios from the suite keep their test
workloads running while waiting for events that might take a long time
to happen, especially on slow or very busy machines.
This patch updates the configuration of the TestWorkload objects to use
a single writer thread (the default is four) in the TestBasicAddition
scenario of the ParameterizedAutoAddMasterTest test. In addition,
a bit of extra delay is now added between writing batches of test rows.
Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Reviewed-on: http://gerrit.cloudera.org:8080/21935
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Marton Greber <gr...@gmail.com>
Reviewed-by: Attila Bukor <ab...@apache.org>
---
M src/kudu/master/dynamic_multi_master-test.cc
1 file changed, 6 insertions(+), 0 deletions(-)
Approvals:
Alexey Serbin: Verified
Marton Greber: Looks good to me, approved
Attila Bukor: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/592/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Oct 2024 22:55:30 +0000
Gerrit-HasComments: No
[kudu-CR] [dynamic multi master-test] reduce the amount of generated data
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21935 )
Change subject: [dynamic_multi_master-test] reduce the amount of generated data
......................................................................
Patch Set 2: Verified+1
unrelated test failures:
* RELEASE: Java tests
* TSAN: Java tests; master-stress-test; subprocess_proxy-test; maintenance_manager-test
--
To view, visit http://gerrit.cloudera.org:8080/21935
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0f0005b8f3384ad05eee6bf65e17a0e4ed289f1
Gerrit-Change-Number: 21935
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Oct 2024 23:55:11 +0000
Gerrit-HasComments: No