You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org> on 2020/07/29 19:12:05 UTC
[kudu-CR] [tool] Add option in loadgen to populate random values only for non-pk columns
Bankim Bhavsar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16253
Change subject: [tool] Add option in loadgen to populate random values only for non-pk columns
......................................................................
[tool] Add option in loadgen to populate random values only for non-pk columns
With --use_random option in loadgen tool, both primary key and non-primary
key columns are populated with random values. This increases
the possibility of collisions for primary key columns when inserting large
number of rows.
This change adds new option --use_random_non_pk that uses random
numbers only for non-primary key columns allowing creation of large
number of rows with random values.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 34 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/1
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 1
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 6
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 Jul 2020 16:55:45 +0000
Gerrit-HasComments: No
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16253
to look at the new patch set (#5).
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
[loadgen] Separate flags to insert random values for PK and non-PK cols
With --use_random option in loadgen tool, both primary key and
non-primary key columns are populated with random values.
This increases the possibility of collisions for primary key columns
when inserting large number of rows.
This change adds two new options:
--use_random_non_pk that uses random numbers only for non-primary key
columns allowing creation of large number of rows with random values
and --use_random_pk that uses random numbers for primary key columns
--use_random is marked as deprecated in the description and if this
option is specified along with --use_random_non_pk or --use_random_pk
then the --use_random will be ignored.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 93 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/5
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 5
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
Patch Set 5: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc@2249
PS5, Line 2249:
nit: fix the indent?
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc@2270
PS5, Line 2270:
nit: fix the indent?
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 5
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 Jul 2020 01:10:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
[loadgen] Separate flags to insert random values for PK and non-PK cols
With --use_random option in loadgen tool, both primary key and
non-primary key columns are populated with random values.
This increases the possibility of collisions for primary key columns
when inserting large number of rows.
This change adds two new options:
--use_random_non_pk that uses random numbers only for non-primary key
columns allowing creation of large number of rows with random values
and --use_random_pk that uses random numbers for primary key columns
--use_random is marked as deprecated in the description and if this
option is specified along with --use_random_non_pk or --use_random_pk
then the --use_random will be ignored.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Reviewed-on: http://gerrit.cloudera.org:8080/16253
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 98 insertions(+), 20 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 7
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16253
to look at the new patch set (#6).
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
[loadgen] Separate flags to insert random values for PK and non-PK cols
With --use_random option in loadgen tool, both primary key and
non-primary key columns are populated with random values.
This increases the possibility of collisions for primary key columns
when inserting large number of rows.
This change adds two new options:
--use_random_non_pk that uses random numbers only for non-primary key
columns allowing creation of large number of rows with random values
and --use_random_pk that uses random numbers for primary key columns
--use_random is marked as deprecated in the description and if this
option is specified along with --use_random_non_pk or --use_random_pk
then the --use_random will be ignored.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 98 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/6
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 6
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [loadgen] Add separate flags to populate random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16253
to look at the new patch set (#4).
Change subject: [loadgen] Add separate flags to populate random values for PK and non-PK cols
......................................................................
[loadgen] Add separate flags to populate random values for PK and non-PK cols
With --use_random option in loadgen tool, both primary key and non-primary
key columns are populated with random values. This increases
the possibility of collisions for primary key columns when inserting large
number of rows.
This change adds 2 new options:
--use_random_non_pk that uses random numbers only for non-primary key columns
allowing creation of large number of rows with random values and
--use_random_pk that uses random numbers for primary key columns
--use_random is marked as deprecated in the description and if this option
is specified along with --use_random_non_pk or --use_random_pk then
the --use_random will be ignored.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 77 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/4
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 4
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [loadgen] Add separate flags to populate random values for PK and non-PK cols
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Add separate flags to populate random values for PK and non-PK cols
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16253/4//COMMIT_MSG
Commit Message:
PS4:
nit: Do you min keeping the lines of the commit description under 72 characters long, when possible? That would be in-line with https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project#_commit_guidelines mentioned at https://kudu.apache.org/docs/contributing.html#_submitting_patches
http://gerrit.cloudera.org:8080/#/c/16253/4//COMMIT_MSG@14
PS4, Line 14: 2
nit: two
http://gerrit.cloudera.org:8080/#/c/16253/4/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/4/src/kudu/tools/kudu-tool-test.cc@2268
PS4, Line 2268: // Test to ensure if both '--use_random' and '--use_random_non_pk'/'--use_random_pk' are
: // specified then '--use_random' is ignored.
: NO_FATALS(RunLoadgen(5,
: {
: // Large number of rows are okay since only non-pk columns will use random values and
: // the deprecated '--use_random' will be ignored when used with '--use_random_non_pk'.
: "--num_rows_per_thread=4096",
: "--num_threads=2",
: "--run_scan",
: "--string_len=8",
: "--use_random",
: "--use_random_non_pk",
: },
: "bench_auto_flush_background_random_deprecated"));
It seems current code assumes there is just a single call of RunLoadgen() per test scenario, otherwise a memory leak happens at https://github.com/apache/kudu/blob/master/src/kudu/integration-tests/cluster_itest_util.cc#L309 called from https://github.com/apache/kudu/blob/master/src/kudu/tools/kudu-tool-test.cc#L913
I guess that's a separate issue, but in this changelist I would simply put those calls under separate test scenarios.
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 4
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 Jul 2020 00:40:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tool] Add option in loadgen to populate random values only for non-pk columns
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16253
to look at the new patch set (#3).
Change subject: [tool] Add option in loadgen to populate random values only for non-pk columns
......................................................................
[tool] Add option in loadgen to populate random values only for non-pk columns
With --use_random option in loadgen tool, both primary key and non-primary
key columns are populated with random values. This increases
the possibility of collisions for primary key columns when inserting large
number of rows.
This change adds 2 new options:
--use_random_non_pk that uses random numbers only for non-primary key columns
allowing creation of large number of rows with random values and
--use_random_pk that uses random numbers for primary key columns
--use_random is marked as deprecated in the description and if this option
is specified along with --use_random_non_pk or --use_random_pk then
the --use_random will be ignored.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 77 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/3
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 3
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [tool] Add option in loadgen to populate random values only for non-pk columns
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [tool] Add option in loadgen to populate random values only for non-pk columns
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc@2240
PS1, Line 2240: "--use_random",
: },
: "bench_auto_flush_bac
> I guess you can safely drop these since it's not essential for the function
Done
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc@2250
PS1, Line 2250: "--buffer_size_bytes=65536",
> nit: I guess it's better to have unique table names even if every test scen
Done
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/tool_action_perf.cc
File src/kudu/tools/tool_action_perf.cc:
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/tool_action_perf.cc@352
PS1, Line 352: "keyspace. Note: The total number of partitions must be greater "
: "than 1.");
: DEFINE_int32(table_num_replicas, 1,
: "The number of replicas for the auto-created table; "
: "0 means 'use server-side default'.");
: DEFINE_bool(use_random, false,
: "Whether to
> What if specifying --use_random=true --use_random_non_pk=false at once? Do
Good suggestion above deprecation.
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 2
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 30 Jul 2020 22:45:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc@2249
PS5, Line 2249:
> nit: fix the indent?
Done. I just let IDE that has Google style activated auto-indent the lines changed in this file.
http://gerrit.cloudera.org:8080/#/c/16253/5/src/kudu/tools/kudu-tool-test.cc@2270
PS5, Line 2270:
> nit: fix the indent?
Done
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 5
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 Jul 2020 16:36:05 +0000
Gerrit-HasComments: Yes
[kudu-CR] [loadgen] Separate flags to insert random values for PK and non-PK cols
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [loadgen] Separate flags to insert random values for PK and non-PK cols
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16253/4//COMMIT_MSG
Commit Message:
PS4:
> nit: Do you min keeping the lines of the commit description under 72 charac
Done
http://gerrit.cloudera.org:8080/#/c/16253/4//COMMIT_MSG@14
PS4, Line 14: t
> nit: two
Done
http://gerrit.cloudera.org:8080/#/c/16253/4/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/4/src/kudu/tools/kudu-tool-test.cc@2268
PS4, Line 2268: "--run_scan",
: "--string_len=8",
: "--use_random_non_pk",
: },
: "bench_auto_flush_background_random_values"));
: }
:
: // Run loadgen benchmark in AUTO_FLUSH_BACKGROUND mode with randomized values
: // using the --use_random_non_pk option combined with the deprecated --use_random option.
: TEST_F(ToolTest, TestLoadgenAutoFlushBackgroundRandomValuesIgnoreDeprecated) {
: // Test to ensure if both '--use_random' and '--use_random_non_pk'/'--use_random_pk' are
: // specified then '--use_random' is ignored.
: NO_FATALS(RunLoadgen(5,
: {
> It seems current code assumes there is just a single call of RunLoadgen() p
Yeah realized that from the ASAN failure logs.
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 5
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 Jul 2020 01:03:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tool] Add option in loadgen to populate random values only for non-pk columns
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16253
to look at the new patch set (#2).
Change subject: [tool] Add option in loadgen to populate random values only for non-pk columns
......................................................................
[tool] Add option in loadgen to populate random values only for non-pk columns
With --use_random option in loadgen tool, both primary key and non-primary
key columns are populated with random values. This increases
the possibility of collisions for primary key columns when inserting large
number of rows.
This change adds new option --use_random_non_pk that uses random
numbers only for non-primary key columns allowing creation of large
number of rows with random values.
Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 34 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/16253/2
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 2
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [tool] Add option in loadgen to populate random values only for non-pk columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/16253 )
Change subject: [tool] Add option in loadgen to populate random values only for non-pk columns
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc@2240
PS1, Line 2240: "--buffer_flush_watermark_pct=0.125",
: "--buffer_size_bytes=65536",
: "--buffers_num=8",
I guess you can safely drop these since it's not essential for the functionality covered by this scenario (randomness of values for non-pk columns only).
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/kudu-tool-test.cc@2250
PS1, Line 2250: bench_auto_flush_background_random
nit: I guess it's better to have unique table names even if every test scenario runs in a separate mini-cluster environment; maybe replace this with 'bench_auto_flush_background_random_non_pk'
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/tool_action_perf.cc
File src/kudu/tools/tool_action_perf.cc:
http://gerrit.cloudera.org:8080/#/c/16253/1/src/kudu/tools/tool_action_perf.cc@352
PS1, Line 352: DEFINE_bool(use_random, false,
: "Whether to use random numbers instead of sequential ones for both primary keys and "
: "non-primary key columns. In case of using random numbers collisions are possible over "
: "the data for columns with unique constraint (e.g. primary key).");
: DEFINE_bool(use_random_non_pk, false,
: "Whether to use random numbers instead of sequential ones for non-primary key "
: "columns.");
What if specifying --use_random=true --use_random_non_pk=false at once? Do we want to warn on such combinations?
From the other side, maybe introduce two new separate flags for pk and non-pk generation mode, and consider '--use_random' deprecated, ignoring its setting if at least one of the newly introduced flags is set to non-default value?
--
To view, visit http://gerrit.cloudera.org:8080/16253
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I35bb33a192baa9e6e67e85bcbd5ca7164ba154e4
Gerrit-Change-Number: 16253
Gerrit-PatchSet: 1
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 29 Jul 2020 20:47:55 +0000
Gerrit-HasComments: Yes