You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2018/06/07 06:13:59 UTC

[kudu-CR] KUDU-1861 add range-partitioning of loadgen tables

Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10633


Change subject: KUDU-1861 add range-partitioning of loadgen tables
......................................................................

KUDU-1861 add range-partitioning of loadgen tables

This patch adds the ability to generate a range-partitioned table to the
laodgen tool. The range partitioning schema is designed such that the
generated insert workload will be monotonically increasing if the number
of threads is equal to the number of tablets.

Below are illustrations of some tablet partitioning and thread
non-random insert workloads. The y-axis for both the threads and the
tablets is the keyspace, increasing going downwards.

--num_threads=2 --table_num_buckets=2

  Threads sequentially
  insert to their keyspaces
  in non-random insert mode.
     +  +---------+         ^
     |  | thread1 | tabletA |  Tablets' range partitions are
     |  |         |         |  set to match the desired total
     v  +---------+---------+  number of inserted rows for the
     |  | thread2 | tabletB |  entire workload, but leaving the
     |  |         |         |  outermost tablets unbounded.
     v  +---------+         v

If the number of tablets is not a multiple of the number of threads when
using an auto-generated range-partitioned table, we lose the guarantee
that we always write to a monotonically increasing range on each tablet.

--num_threads=2 --table_num_buckets=3
     +  +---------+         ^
     |  | thread1 | tabletA |
     |  |         +---------+
     v  +---------| tabletB |
     |  | thread2 +---------+
     |  |         | tabletC |
     v  +---------+         v

I tested this out on a singler-tserver cluster and verified via the
metrics logs that the number of bloom lookups for a non-random workload
where the number of insert threads and the number of tablets were equal
stayed at zero. When the number of threads was not a factor of the
number of buckets, the number of bloom lookups was non-zero.

Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
---
M src/kudu/tools/tool_action_perf.cc
1 file changed, 36 insertions(+), 8 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/33/10633/1
-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.

Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10633

to look at the new patch set (#4).

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................

KUDU-1861: add range-partitions to loadgen tables

This patch adds the ability to generate a range-partitioned table with
the loadgen tool. The range partitioning schema is designed such that
the non-random write workload will insert sequentially on the primary
key, provided the number of threads is equal to the number of tablets.
This sequential workload per tablet both reduces the number of
compactions and avoids bloom filter lookups.

This patch also renames --table_num_buckets to
--table_num_hash_partitions, which can be combined with
--table_num_range_partitions if desired.

I tested this out on a singler-tserver cluster and verified via the
metrics logs that the number of bloom lookups for a non-random workload
where the number of insert threads and the number of tablets were equal
stayed at zero. When the number of threads was not a factor of the
number of buckets, the number of bloom lookups was non-zero. See
TestNonRandomWorkloadLoadgen in kudu-tool-test.cc for more details.

Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 229 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/33/10633/4
-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10633 )

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................


Patch Set 6:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc@1487
PS5, Line 1487: 
              :     // Keep the tab
> Actually I did mean to use `use_random` to make sure nothing break _even if
Thank you for the explanation.  Could you add those details into the comment itself?  I think it would be clearer for the reader than '... nothing funny happens ...'.


http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc@1512
PS5, Line 1512:   args.emplace_back("--table_num_range_partitions=2");
              :   args.emplace_back("--table_num_hash_partitio
> We're not deleting the tables as we go so we can do this check. That also m
I'm not sure I understand.  My comment was about reducing

ASSERT_OK(WaitForNumTabletsOnTS(ts, expected_tablets, kTimeout, &tablets));
ASSERT_EQ(expected_tablets, tablets.size());

to

SSERT_OK(WaitForNumTabletsOnTS(ts, expected_tablets, kTimeout));

The idea is that you don't need the second assert because of the way how WaitForNumTabletsOnTS() works.


http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc
File src/kudu/tools/tool_action_perf.cc:

http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc@325
PS6, Line 325: static 
nit: why do you need the 'static' specifier for the function within an anonymous namespace?



-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Sat, 23 Jun 2018 04:14:07 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.

Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10633

to look at the new patch set (#5).

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................

KUDU-1861: add range-partitions to loadgen tables

This patch adds the ability to generate a range-partitioned table with
the loadgen tool. The range partitioning schema is designed such that
the non-random write workload will insert sequentially on the primary
key, provided the number of threads is equal to the number of tablets.
This sequential workload per tablet both reduces the number of
compactions and avoids bloom filter lookups.

This patch also renames --table_num_buckets to
--table_num_hash_partitions, which can be combined with
--table_num_range_partitions if desired.

I tested this out on a singler-tserver cluster and verified via the
metrics logs that the number of bloom lookups for a non-random workload
where the number of insert threads and the number of tablets were equal
stayed at zero. When the number of threads was not a factor of the
number of buckets, the number of bloom lookups was non-zero. See
TestNonRandomWorkloadLoadgen in kudu-tool-test.cc for more details.

Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 234 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/33/10633/5
-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.

Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10633 )

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................

KUDU-1861: add range-partitions to loadgen tables

This patch adds the ability to generate a range-partitioned table with
the loadgen tool. The range partitioning schema is designed such that
the non-random write workload will insert sequentially on the primary
key, provided the number of threads is equal to the number of tablets.
This sequential workload per tablet both reduces the number of
compactions and avoids bloom filter lookups.

This patch also renames --table_num_buckets to
--table_num_hash_partitions, which can be combined with
--table_num_range_partitions if desired.

I tested this out on a singler-tserver cluster and verified via the
metrics logs that the number of bloom lookups for a non-random workload
where the number of insert threads and the number of tablets were equal
stayed at zero. When the number of threads was not a factor of the
number of buckets, the number of bloom lookups was non-zero. See
TestNonRandomWorkloadLoadgen in kudu-tool-test.cc for more details.

Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Reviewed-on: http://gerrit.cloudera.org:8080/10633
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 244 insertions(+), 16 deletions(-)

Approvals:
  Kudu Jenkins: Verified
  Alexey Serbin: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 9
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.

Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10633 )

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................


Patch Set 7:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc@1487
PS5, Line 1487: 
              :     // Keep the tab
> Thank you for the explanation.  Could you add those details into the commen
Done


http://gerrit.cloudera.org:8080/#/c/10633/5/src/kudu/tools/kudu-tool-test.cc@1512
PS5, Line 1512:   args = base_args;
              :   args.emplace_back("--table_num_range_partiti
> I'm not sure I understand.  My comment was about reducing
Ah my bad, I misunderstood. Done


http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc
File src/kudu/tools/tool_action_perf.cc:

http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc@197
PS6, Line 197: #include "kudu/util/flag_validators.h"
> warning: #includes are not sorted properly [llvm-include-order]
Done


http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc@325
PS6, Line 325: 
> nit: why do you need the 'static' specifier for the function within an anon
Done


http://gerrit.cloudera.org:8080/#/c/10633/6/src/kudu/tools/tool_action_perf.cc@325
PS6, Line 325: 
> warning: 'ValidatePartitionFlags' is a static definition in anonymous names
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Sat, 23 Jun 2018 05:29:01 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1861: add range-partitions to loadgen tables

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.

Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10633

to look at the new patch set (#8).

Change subject: KUDU-1861: add range-partitions to loadgen tables
......................................................................

KUDU-1861: add range-partitions to loadgen tables

This patch adds the ability to generate a range-partitioned table with
the loadgen tool. The range partitioning schema is designed such that
the non-random write workload will insert sequentially on the primary
key, provided the number of threads is equal to the number of tablets.
This sequential workload per tablet both reduces the number of
compactions and avoids bloom filter lookups.

This patch also renames --table_num_buckets to
--table_num_hash_partitions, which can be combined with
--table_num_range_partitions if desired.

I tested this out on a singler-tserver cluster and verified via the
metrics logs that the number of bloom lookups for a non-random workload
where the number of insert threads and the number of tablets were equal
stayed at zero. When the number of threads was not a factor of the
number of buckets, the number of bloom lookups was non-zero. See
TestNonRandomWorkloadLoadgen in kudu-tool-test.cc for more details.

Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_perf.cc
2 files changed, 244 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/33/10633/8
-- 
To view, visit http://gerrit.cloudera.org:8080/10633
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4f552a4c73dc82f3b7934082769522557ee5013
Gerrit-Change-Number: 10633
Gerrit-PatchSet: 8
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot