You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org> on 2021/06/01 23:26:52 UTC
[Impala-ASF-CR] IMPALA-7501: Slim down partition metadata in LocalCatalog mode
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17505 )
Change subject: IMPALA-7501: Slim down partition metadata in LocalCatalog mode
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@356
PS2, Line 356: // ... each such partition should include the HMS partition struct.
: 12: bool want_hms_partition
nit, move to the end so that field numbers are in sequence?
http://gerrit.cloudera.org:8080/#/c/17505/2/common/thrift/CatalogService.thrift@397
PS2, Line 397: // Fields 10-13 are set if 'want_partition_metadata' was set in TTableInfoSelector.
: // These fields are actual info of hms_partition that Impala needs, and are better
: // compressed.
: 10: optional map<string, string> hms_parameters
: 11: optional i64 write_id
: 12: optional CatalogObjects.THdfsStorageDescriptor hdfs_storage_descriptor
: 13: optional CatalogObjects.THdfsPartitionLocation location
can we move them to the end so that field numbers are in sequence.
http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@357
PS4, Line 357: 12: bool want_hms_partition
IMO, it is less error-prone to keep this field after field 11. You can add a comment about the difference between want_partition_metadata and want_hms_partition to make it more readable after moving this line below.
http://gerrit.cloudera.org:8080/#/c/17505/4/common/thrift/CatalogService.thrift@397
PS4, Line 397: // Fields 10-13 are set if 'want_partition_metadata' was set in TTableInfoSelector.
: // These fields are actual info of hms_partition that Impala needs, and are better
: // compressed.
: 10: optional map<string, string> hms_parameters
: 11: optional i64 write_id
: 12: optional CatalogObjects.THdfsStorageDescriptor hdfs_storage_descriptor
: 13: optional CatalogObjects.THdfsPartitionLocation location
Can we move them to after line 428. Otherwise it may become error-prone if someone adds a new field to this struct and reuses the field identifiers.
http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/17505/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@960
PS4, Line 960: hdfsStorageDescriptor
It looks like we are sending the HdfsStorageDescriptor for each partition here. We just changed it into a more optimized form. Is it possible to send only one copy of it at the table level and then construct it from the table's storageDescriptor like what catalogd does?
--
To view, visit http://gerrit.cloudera.org:8080/17505
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I307e7a8193b54a7b3ab93d9ebd194766bbdbd977
Gerrit-Change-Number: 17505
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 23:26:52 +0000
Gerrit-HasComments: Yes