You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2021/08/12 00:28:40 UTC
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17769
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys if range partitions with custom hash bucket schemas are present.
I added a few new test scenarios and updated the existing ones
accordingly.
In addition, I sneaking in minor style and formatting updates in
* src/kudu/tablet/tablet.cc
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
11 files changed, 249 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/17769/1
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(4 comments)
Thank you for the feedback!
I'm about to address it in a separate changelist. As of now, I just addressed a few questions.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> What's the benefit of storing the table wide hash schema for each range tha
Right: eventually there should be no table-wide hash schema stored. There will be a registry of all different hash schemas used by a table's partition schema, and then each range will have a reference (i.e. pointer/index/iterator) pointing to corresponding element in the registry.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@959
PS6, Line 959: if (key.size() < kEncodedBucketSize * hash_bucket_schemas_.size()) {
: return "<hash-decode-error>";
: }
> Should we check this for per range hash schemas as well?
Good point! I guess this part should have been moved down right before the for() cycle at line 975.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1558
PS6, Line 1558: // TODO(aserbin): is the upper bound always exclusive?
> yes, lower bound is always inclusive and upper bound is always exclusive.
Thank you for the confirmation -- I removed this comment in https://gerrit.cloudera.org/#/c/17775/
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> Maybe I'm missing something, but why would this if statement ever be trigge
This statement is triggered in a case of non-covered range: FindFloorOrNull() returns a range with its lower boundary less or equal to the specified range key, but it might happen that even the upper boundary of the range less than the key.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 04:38:37 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Mahesh Reddy, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17769
to look at the new patch set (#2).
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys in the presence of range partitions with custom hash bucket schemas
in a Kudu table. I added a few new test scenarios and updated the
existing ones accordingly to make it possible to send write operations
into appropriate tablets.
In addition, I sneaking in minor style and formatting updates in
* src/kudu/tablet/tablet.cc
* src/kudu/master/catalog_manager.cc
As one can see, the newly added method GetHashBucketSchemasForKey() is
used in PartitionSchema also in case when Partition object is at hand.
A follow-up changelist is going to change that, so there will be no need
to perform a dictionary lookup once the Partition class contains
information about its hash bucket schema.
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
11 files changed, 249 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/17769/2
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Mahesh Reddy, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17769
to look at the new patch set (#5).
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys in the presence of range partitions with custom hash bucket schemas
in a Kudu table. I added a few new test scenarios and updated the
existing ones accordingly to make it possible to send write operations
into appropriate tablets.
In addition, I'm sneaking in minor style-related updates in
* src/kudu/tablet/tablet.cc
* src/kudu/master/catalog_manager.cc
As one can see, the newly added method GetHashBucketSchemasForKey() is
used in PartitionSchema also in case when Partition object is at hand.
A follow-up changelist is going to change that, so there will be no need
to perform a dictionary lookup once the Partition class contains
information about its hash bucket schema.
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition-test.cc
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
12 files changed, 340 insertions(+), 107 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/17769/5
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Mahesh Reddy, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17769
to look at the new patch set (#3).
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys in the presence of range partitions with custom hash bucket schemas
in a Kudu table. I added a few new test scenarios and updated the
existing ones accordingly to make it possible to send write operations
into appropriate tablets.
In addition, I'm sneaking in minor style-related updates in
* src/kudu/tablet/tablet.cc
* src/kudu/master/catalog_manager.cc
As one can see, the newly added method GetHashBucketSchemasForKey() is
used in PartitionSchema also in case when Partition object is at hand.
A follow-up changelist is going to change that, so there will be no need
to perform a dictionary lookup once the Partition class contains
information about its hash bucket schema.
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
11 files changed, 249 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/17769/3
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> The only issue with this approach is that the user needs to manually add a
Nope, there is no need to require user to manually add a hash dimension for each range. That's just how the data will be represented internally.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 19:47:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc@228
PS4, Line 228: shared_ptr<KuduTable> table;
: ASSERT_OK(client_->OpenTable(table_name, &table));
:
: vector<scoped_refptr<TabletInfo>> all_tablets_info;
: {
: auto* cm = cluster_->mini_master(0)->master()->catalog_manager();
: CatalogManager::ScopedLeaderSharedLock l(cm);
: scoped_refptr<master::TableInfo> table_info;
: ASSERT_OK(cm->GetTableInfo(table->id(), &table_info));
: table_info->GetAllTablets(&all_tablets_info);
: }
: vector<scoped_refptr<TabletReplica>> replicas;
: for (const auto& tablet_info : all_tablets_info) {
: for (auto i = 0; i < cluster_->num_tablet_servers(); ++i) {
: scoped_refptr<TabletReplica> r;
: ASSERT_TRUE(cluster_->mini_tablet_server(i)->server()->
: tablet_manager()->LookupTablet(tablet_info->id(), &r));
: replicas.emplace_back(std::move(r));
: }
: }
:
: int64_t count = 0;
: for (const auto& r : replicas) {
: count += r->CountLiveRowsNoFail();
: }
> nit: would client_->GetTableStatistics() work instead?
Yeah, but that's super slow -- even with 10ms tablet server to master hearbeat interval, it takes many retries under ASSERT_EVENTUALLY() to get this reported as needed. Compare:
FlexPartitioningCreateTableTest.DefaultAndCustomHashBuckets (10261 ms)
vs
FlexPartitioningCreateTableTest.DefaultAndCustomHashBuckets (1007 ms)
I guess at some point we will remove this chunk of code anyways once partition pruning works, so maybe let's strive to keep the testing time lower until the partition pruning is correctly implemented?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1525
PS3, Line 1525:
> I see. I suppose an alternative would be
Yeah, that would make a very long string. I think I'd rather revamp the terminology throughout the code here in a separate changelist:
* hash bucket schema --> hash schema
* hash partitioning schema --> hash schema
* range partition schema --> range schema
With that, the naming would be more universal and there should be opportunities to shorten some lines in the code.
Does it make sense to you?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1528
PS3, Line 1528:
> Thanks for fixing that! It's much clearer now.
Thank you for pointing to the inconsistency -- it seems this class uses a clean-up on terminology used for various entities.
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc@260
PS4, Line 260: std::distance(ranges_ptr->cbegin(), it)
> Rather than pointing to an index, would it make sense to have the value be
It might be an option and I started with that (well, I did use iterators to be able to spot those issues I described below), but then I realized it would be necessary to prohibit copying and moving instances of PartitionSchema or implement custom copying & moving operators. Otherwise there would be dangling pointers -- as of now, there are use cases in the code when PartionSchema objects are being copied or moved.
I'd rather clean up the code in that regard and introduce custom copying/moving operators in a separate changelist, moving this into using iterators/pointers instead.
Is it OK with you?
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc@1547
PS4, Line 1547: key
> nit: range_key
Done
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Aug 2021 19:40:25 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> Makes sense, when we get rid of the renamed field 'hash_schema_', what will
The return value should be the appropriate hash schema corresponding to the range, regardless of the internal representation of the hash schema information in this class.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 19:50:41 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
What's the benefit of storing the table wide hash schema for each range that uses it rather than using an empty index to signify the usage of the table wide hash schema? I guess if eventually if we want to phase out the use of the newly renamed 'hash_schema field' within PartitionSchema then this change makes sense.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc@382
PS6, Line 382: the
remove 'the'
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc@382
PS6, Line 382: paritions
nit: partitions
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.h@499
PS6, Line 499: key
nit: range key
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@959
PS6, Line 959: if (key.size() < kEncodedBucketSize * hash_bucket_schemas_.size()) {
: return "<hash-decode-error>";
: }
Should we check this for per range hash schemas as well?
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1558
PS6, Line 1558: // TODO(aserbin): is the upper bound always exclusive?
yes, lower bound is always inclusive and upper bound is always exclusive.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
Maybe I'm missing something, but why would this if statement ever be triggered?
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Aug 2021 21:54:14 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc@228
PS4, Line 228: shared_ptr<KuduTable> table;
: ASSERT_OK(client_->OpenTable(table_name, &table));
:
: vector<scoped_refptr<TabletInfo>> all_tablets_info;
: {
: auto* cm = cluster_->mini_master(0)->master()->catalog_manager();
: CatalogManager::ScopedLeaderSharedLock l(cm);
: scoped_refptr<master::TableInfo> table_info;
: ASSERT_OK(cm->GetTableInfo(table->id(), &table_info));
: table_info->GetAllTablets(&all_tablets_info);
: }
: vector<scoped_refptr<TabletReplica>> replicas;
: for (const auto& tablet_info : all_tablets_info) {
: for (auto i = 0; i < cluster_->num_tablet_servers(); ++i) {
: scoped_refptr<TabletReplica> r;
: ASSERT_TRUE(cluster_->mini_tablet_server(i)->server()->
: tablet_manager()->LookupTablet(tablet_info->id(), &r));
: replicas.emplace_back(std::move(r));
: }
: }
:
: int64_t count = 0;
: for (const auto& r : replicas) {
: count += r->CountLiveRowsNoFail();
: }
nit: would client_->GetTableStatistics() work instead?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1525
PS3, Line 1525:
> Well, instead I suggest to treat the previous line same as if it were
I see. I suppose an alternative would be
const PartitionSchema::HashBucketSchemas& PartitionSchema::GetHashBucketSchemasForKey(
const string& key) const {
which is more in line with style elsewhere. That said, I don't feel strongly about it, given I agree the length of the types makes it tougher to read.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1528
PS3, Line 1528:
> You are right: the comment for the method is incorrect -- that's the range
Thanks for fixing that! It's much clearer now.
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc@260
PS4, Line 260: std::distance(ranges_ptr->cbegin(), it)
Rather than pointing to an index, would it make sense to have the value be the corresponding RangeWithHashSchemas*?
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc@1547
PS4, Line 1547: key
nit: range_key
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Aug 2021 00:37:46 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> The Kudu API will be backward-compatible, so the user could still be able t
Ah I see, but within the PartitionSchema class and PartitionSchemaPB we will phase out 'hash_schema_'. So to maintain backwards compatibility within the API, the user can still define the table wide hash schema but when populating PartitionSchemaPB and eventually the PartitionSchema class we will only store the ranges and their respective hash schemas.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> If there is no range for the key, then the method will let the caller know
Ack
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 22:07:30 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> Ack
Alternatively, it's possible to think about the table-wide hash schema as always present, but it can contains zero hash dimensions (e.g. if no hash partitioning is there at all).
However, we should phase out the PartitionSchema::hash_schema() method (here I'm using the notations after the recent renamings).
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 22:38:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Mahesh Reddy, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17769
to look at the new patch set (#4).
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys in the presence of range partitions with custom hash bucket schemas
in a Kudu table. I added a few new test scenarios and updated the
existing ones accordingly to make it possible to send write operations
into appropriate tablets.
In addition, I'm sneaking in minor style-related updates in
* src/kudu/tablet/tablet.cc
* src/kudu/master/catalog_manager.cc
As one can see, the newly added method GetHashBucketSchemasForKey() is
used in PartitionSchema also in case when Partition object is at hand.
A follow-up changelist is going to change that, so there will be no need
to perform a dictionary lookup once the Partition class contains
information about its hash bucket schema.
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition-test.cc
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
12 files changed, 341 insertions(+), 107 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/17769/4
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 3:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/client.cc@1017
PS3, Line 1017:
> nit: got a couple extra spaces here.
Done
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/flex_partitioning_client-test.cc@328
PS3, Line 328: 44));
:
:
> We may not be able to scan the table yet, but perhaps we can check the tabl
Good point. Done.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h@510
PS3, Line 510: // NOTE: this map becomes invalid once 'ranges_with_hash_schemas_' changed
> nit: rather than mentioning when it's valid, maybe mention that users shoul
Done
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h@512
PS3, Line 512: hash_schema_
> nit: maybe name this and the typedef hash_schema_idx_by... or something, so
Done
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@260
PS3, Line 260: Arena arena(256);
: Slice s_lower = Slice(it->lower);
: KuduPartialRow lower(&schema);
: RETURN_NOT_OK(partition_schema->DecodeRangeKey(&s_lower, &lower, &arena));
> I think I'm missing something -- why do we need to decode the range key if
Ah, good catch -- not sure how I missed this.
That must be a result of conflict resolution or something while squashing commits and re-basing: this piece doesn't effectively do anything to affect it->lower, so it's a no-op :)
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1525
PS3, Line 1525: Part
> nit: add four spaces since this is a continuation of the previous line?
Well, instead I suggest to treat the previous line same as if it were
template<typename T>
Just discovered that making those continuations between the return type and the method name look way off and bring more puzzle than readability :)
I can change this if you feel strongly about changing the formatting as suggested, though.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1528
PS3, Line 1528: hash_schema_by_encoded_range_start_, key
> If the input 'key' is a partition key, how can we use it as a key to a map
You are right: the comment for the method is incorrect -- that's the range key, of course. Here I used 'partition key' notion in meaning of ranges in one linear dimension.
I fixed this terminology issue.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Aug 2021 08:54:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 5: Code-Review+2
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/client/flex_partitioning_client-test.cc@228
PS4, Line 228: shared_ptr<KuduTable> table;
: ASSERT_OK(client_->OpenTable(table_name, &table));
:
: vector<scoped_refptr<TabletInfo>> all_tablets_info;
: {
: auto* cm = cluster_->mini_master(0)->master()->catalog_manager();
: CatalogManager::ScopedLeaderSharedLock l(cm);
: scoped_refptr<master::TableInfo> table_info;
: ASSERT_OK(cm->GetTableInfo(table->id(), &table_info));
: table_info->GetAllTablets(&all_tablets_info);
: }
: vector<scoped_refptr<TabletReplica>> replicas;
: for (const auto& tablet_info : all_tablets_info) {
: for (auto i = 0; i < cluster_->num_tablet_servers(); ++i) {
: scoped_refptr<TabletReplica> r;
: ASSERT_TRUE(cluster_->mini_tablet_server(i)->server()->
: tablet_manager()->LookupTablet(tablet_info->id(), &r));
: replicas.emplace_back(std::move(r));
: }
: }
:
: int64_t count = 0;
: for (const auto& r : replicas) {
: count += r->CountLiveRowsNoFail();
: }
> Yeah, but that's super slow -- even with 10ms tablet server to master hearb
Yep, sgtm. Makes sense, given this is all actively being developed.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@260
PS3, Line 260: InsertOrDie(&dict, it->lower, std::distance(ranges_ptr->cbegin(), it));
: }
: }
: if (range_bounds.size() != ranges_ptr->size()) {
> Ah, good catch -- not sure how I missed this.
Ack
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1525
PS3, Line 1525:
> Yeah, that would make a very long string. I think I'd rather revamp the te
Sounds good. I agree it makes sense to consolidate terminology, especially if it helps reduce length without sacrificing clarity.
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/4/src/kudu/common/partition.cc@260
PS4, Line 260: std::distance(ranges_ptr->cbegin(), it)
> It might be an option and I started with that (well, I did use iterators to
Ah I see. Yeah I'm fine with leaving this as is for this patch.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Aug 2021 20:21:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 3:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/client.cc@1017
PS3, Line 1017:
nit: got a couple extra spaces here.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/client/flex_partitioning_client-test.cc@328
PS3, Line 328: 44));
:
:
We may not be able to scan the table yet, but perhaps we can check the table stats and get the number of live rows?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h@510
PS3, Line 510: // NOTE: this map becomes invalid once 'ranges_with_hash_schemas_' changed
nit: rather than mentioning when it's valid, maybe mention that users should clear/set this as ranges_with_hash_schemas_ is cleared and set? I found myself wondering when 'ranges_with_hash_schemas_' could be mutated, and if so, how we ensure this wasn't used. But it seems the usage of these two are fairly tightly coupled.
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.h@512
PS3, Line 512: hash_schema_
nit: maybe name this and the typedef hash_schema_idx_by... or something, so it's clear at usage sites these refer to an index?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@260
PS3, Line 260: Arena arena(256);
: Slice s_lower = Slice(it->lower);
: KuduPartialRow lower(&schema);
: RETURN_NOT_OK(partition_schema->DecodeRangeKey(&s_lower, &lower, &arena));
I think I'm missing something -- why do we need to decode the range key if we're keying by the encoded range key?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1525
PS3, Line 1525: Part
nit: add four spaces since this is a continuation of the previous line?
http://gerrit.cloudera.org:8080/#/c/17769/3/src/kudu/common/partition.cc@1528
PS3, Line 1528: hash_schema_by_encoded_range_start_, key
If the input 'key' is a partition key, how can we use it as a key to a map that's keyed by range key?
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Aug 2021 03:35:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> Nope, there is no need to require user to manually add a hash dimension for
Ah, I guess your point was about the PB structures (which are populated here). If that's what you meant here, I totally agree -- we might optimize this piece, having some sort of agreement how to interpret the elements in the sequence of (range + hash schema) elements. Anyway, I'm planning to update this code, at least to allow for having no hash-based sharding for partitions, when needed.
But overall, the internal representation of the range-specific hash schemas and how we populate the PB structures for RPC requests might be different: we should give the client side the ability to be more concise, while we might strive for faster lookup of range-specific schema in the client library.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> To be clear, I was referring to when the table-wide hash schema would be ph
The Kudu API will be backward-compatible, so the user could still be able to use the KuduTableCreator::add_hash_partitions() call which is setting table-wide schema. So, no: users of the Kudu API will not need to add hash dimensions for each range -- they might opt out to call KuduTableCreator::add_hash_partitions() to specify table-wide schema and then add range partitions with custom hash schemas using KuduRangePartition::add_hash_partitions().
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc
File src/kudu/client/flex_partitioning_client-test.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc@382
PS6, Line 382: paritions
> nit: partitions
Done
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/flex_partitioning_client-test.cc@382
PS6, Line 382: the
> remove 'the'
Done
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.h@499
PS6, Line 499: key
> nit: range key
Done
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> Right, but what will be returned in the case of a non-covered range once th
If there is no range for the key, then the method will let the caller know about such a condtition. Of course, it might require changing the function's signature :)
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 21:15:48 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
KUDU-2671 key encoding for custom hash bucket schemas
This patch updates the PartitionSchema class to properly encode range
keys in the presence of range partitions with custom hash bucket schemas
in a Kudu table. I added a few new test scenarios and updated the
existing ones accordingly to make it possible to send write operations
into appropriate tablets.
In addition, I'm sneaking in minor style-related updates in
* src/kudu/tablet/tablet.cc
* src/kudu/master/catalog_manager.cc
As one can see, the newly added method GetHashBucketSchemasForKey() is
used in PartitionSchema also in case when Partition object is at hand.
A follow-up changelist is going to change that, so there will be no need
to perform a dictionary lookup once the Partition class contains
information about its hash bucket schema.
This is a follow-up to 586b7913258df2d0ee75470ddfb2b88d472ba235.
Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Reviewed-on: http://gerrit.cloudera.org:8080/17769
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/flex_partitioning_client-test.cc
M src/kudu/client/meta_cache.cc
M src/kudu/client/write_op.h
M src/kudu/common/key_encoder.cc
M src/kudu/common/key_encoder.h
M src/kudu/common/partition-test.cc
M src/kudu/common/partition.cc
M src/kudu/common/partition.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tablet/tablet.cc
12 files changed, 340 insertions(+), 107 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> Right: eventually there should be no table-wide hash schema stored. There
The only issue with this approach is that the user needs to manually add a hash dimension for each range, even if the same hash dimension is to be used for each range or most of the ranges. That would put more of an onus on the user.
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@959
PS6, Line 959: if (key.size() < kEncodedBucketSize * hash_bucket_schemas_.size()) {
: return "<hash-decode-error>";
: }
> Good point! I guess this part should have been moved down right before the
Ack
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1558
PS6, Line 1558: // TODO(aserbin): is the upper bound always exclusive?
> Thank you for the confirmation -- I removed this comment in https://gerrit.
Ack
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> This statement is triggered in a case of non-covered range: FindFloorOrNull
Makes sense, when we get rid of the renamed field 'hash_schema_', what will be returned if this if condition is met?
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 08:41:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> Ah I see, but within the PartitionSchema class and PartitionSchemaPB we wil
Right -- in the implementation we can optimize for the use-case of custom hash schema per range or whatever we want. But we have to keep the Kudu client API backward compatible, so the apps written before adding per-range hash schema can be built/linked with the newer Kudu library.
Also, the older clients should be able to work with the new servers, so we have to keep the notion of the table-wide schema at PartitionSchemaPB as is, and interpret it as needed.
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 22:25:39 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2671 key encoding for custom hash bucket schemas
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17769 )
Change subject: KUDU-2671 key encoding for custom hash bucket schemas
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc
File src/kudu/client/client.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/client/client.cc@1016
PS6, Line 1016: if (range->hash_bucket_schemas_.empty()) {
: schemas_pb->mutable_hash_schemas()->CopyFrom(
: data_->partition_schema_.hash_bucket_schemas())
> Nope, there is no need to require user to manually add a hash dimension for
To be clear, I was referring to when the table-wide hash schema would be phased out. Then, the user would have to manually add hash dimensions for each range right?
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc
File src/kudu/common/partition.cc:
http://gerrit.cloudera.org:8080/#/c/17769/6/src/kudu/common/partition.cc@1559
PS6, Line 1559: if (!upper.empty() && upper <= range_key) {
: has_custom_range = false;
: }
> The return value should be the appropriate hash schema corresponding to the
Right, but what will be returned in the case of a non-covered range once the table-wide hash schema is phased out?
--
To view, visit http://gerrit.cloudera.org:8080/17769
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I81aa1c10998d88a1bd5314fc3132037b1c8bbe4b
Gerrit-Change-Number: 17769
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Aug 2021 20:54:16 +0000
Gerrit-HasComments: Yes