You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tamas Mate (Code Review)" <ge...@cloudera.org> on 2022/02/11 15:10:55 UTC
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18226
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Iceberg stores the table metadata next to the data files, when this is
accessed through the Iceberg API a filesystem call is executed (HDFS,
S3, ADLS). This patch moves the individual Iceberg API load calls to the
CatalogD and replaces them with metatada requests from the CatalogD.
Impala daemons no longer load files from the filesystem and the Catalog
daemon caches the Iceberg metadata similar to other tables. As a result
REFRESH/INVALIDATE METADATA is required to reload the Iceberg metadata.
This improvement is more significant for smaller queries, where the
metadata request has larger impact on the query time.
Testing:
- Passed Iceberg E2E tests.
Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
---
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
16 files changed, 215 insertions(+), 151 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18226/1
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1948
PS9, Line 1948: private static class IcebergMetaCacheKey extends VersionedTableCacheKey {
This calculates the same hash value as PartitionListCacheKey (and VersionedTableCacheKey).
We don't invoke loadPartitionList() for Iceberg tables so it doesn't cause a problem, but maybe it'd be safer to override hashCode(). Because some other load method might also start to use VersionedTableCacheKey and its hashCode() method.
So how about e.g.:
@Override
public int hashCode() {
return Objects.hashCode(super.hashCode(), getClass());
}
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 11:53:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10249/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 08:00:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9: Code-Review+2
(1 comment)
Looks great!
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1948
PS9, Line 1948: private static class IcebergMetaCacheKey extends VersionedTableCacheKey {
> VersionedTableCacheKey inherits similar hash function, with dbname/tablenam
Thanks for the clarification!
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 15:24:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
The reason of the failure was that the LocalTable.load() has already loaded and cached some metadata as TableMetaRefImp, while the LocalIcebergTable was trying to retrieve TPartialTableInfo. Added a new CacheKey type to cache the Iceberg metadata and Table metadata separately.
Additionally, I had to fix the 'loadFileDescMapFromThrift' parameters.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 10:01:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7901/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 07:47:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10244/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 10:20:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@500
PS1, Line 500: ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(serializedBaseTable);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@437
PS1, Line 437: icebergBaseTable_ = FeIcebergTable.Utils.deserializeIcebergBaseTable(ticeberg.getIceberg_base_table());
line too long (107 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Feb 2022 15:11:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 1:
(1 comment)
The change looks good. I only had one ask about caching which was overlooked earlier.
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1036
PS1, Line 1036: TGetPartialCatalogObjectResponse resp = sendRequest(req);
Currently we always send the request for every query. Can we add caching? See loadWithCaching() above.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Feb 2022 12:04:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7900/
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 21:59:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7899/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 10:01:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18226/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1948
PS9, Line 1948: private static class IcebergMetaCacheKey extends VersionedTableCacheKey {
> This calculates the same hash value as PartitionListCacheKey (and Versioned
VersionedTableCacheKey inherits similar hash function, with dbname/tablename and getClass(), from DbChildCacheKey through TableCacheKey, this should be unique for IcebergMetadata.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 15:08:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7900/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 17:13:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9:
The latest build appears to be a flakiness problem as well, test_acid_compute_stats failed on this comparison:
assert expected_results == actual_results
E assert Comparing QueryTestResults (expected vs actual):
E '1',1,1,'2B','NOT CACHED','NOT CACHED',regex:.*,'true',regex:.*
!= '1',1,1,'2B','NOT CACHED','NOT CACHED','TEXT','false','hdfs://<truncated>/test_acid_compute_stats_69ccf940.db/pt/p=1'
E 'Total',1,1,'2B','0B','','','','' == 'Total',1,1,'2B','0B','','','',''
Looks like the diff is in the isIncrementalStats column.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 07:46:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18226/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1036
PS1, Line 1036: TGetPartialCatalogObjectResponse resp = sendRequest(req);
> Currently we always send the request for every query. Can we add caching? S
Updated this part with loadWithCaching().
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 12:18:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Iceberg stores the table metadata next to the data files, when this is
accessed through the Iceberg API a filesystem call is executed (HDFS,
S3, ADLS). These calls were used in various places during query
processing and this patch unifies the Iceberg metadata request in the
CatalogD similar to other metadata requests:
- CatalogD loads and caches the org.apache.iceberg.BaseTable object.
- ImpalaDs requests the org.apache.iceberg.BaseTable from the
CatalogD and caches it as well.
As a result REFRESH/INVALIDATE METADATA is required to reload any
Iceberg metadata changes and the metadata load time is improved.
This improvement is more significant for smaller queries, where the
metadata request has larger impact on the query execution time.
Testing:
- Passed Iceberg E2E tests.
Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
---
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
17 files changed, 230 insertions(+), 175 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18226/9
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18226/8/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/18226/8/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@84
PS8, Line 84: TPartialTableInfo tableInfo = db.getCatalog().getMetaProvider().loadIcebergTable(ref);
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 10:01:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8:
I believe this failure was caused by test_async_load flakiness (IMPALA-10999), I am restarting the build.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 17:12:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7899/
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 14:45:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10143/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Feb 2022 15:33:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Iceberg stores the table metadata next to the data files, when this is
accessed through the Iceberg API a filesystem call is executed (HDFS,
S3, ADLS). These calls were used in various places during query
processing and this patch unifies the Iceberg metadata request in the
CatalogD similar to other metadata requests:
- CatalogD loads and caches the org.apache.iceberg.BaseTable object.
- ImpalaDs requests the org.apache.iceberg.BaseTable from the
CatalogD and caches it as well.
As a result REFRESH/INVALIDATE METADATA is required to reload any
Iceberg metadata changes and the metadata load time is improved.
This improvement is more significant for smaller queries, where the
metadata request has larger impact on the query execution time.
Testing:
- Passed Iceberg E2E tests.
Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
---
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
17 files changed, 229 insertions(+), 175 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18226/8
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 8
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 9
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 12:44:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests
......................................................................
IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Iceberg stores the table metadata next to the data files, when this is
accessed through the Iceberg API a filesystem call is executed (HDFS,
S3, ADLS). These calls were used in various places during query
processing and this patch unifies the Iceberg metadata request in the
CatalogD similar to other metadata requests:
- CatalogD loads and caches the org.apache.iceberg.BaseTable object.
- ImpalaDs requests the org.apache.iceberg.BaseTable from the
CatalogD and caches it as well.
As a result REFRESH/INVALIDATE METADATA is required to reload any
Iceberg metadata changes and the metadata load time is improved.
This improvement is more significant for smaller queries, where the
metadata request has larger impact on the query execution time.
Testing:
- Passed Iceberg E2E tests.
Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Reviewed-on: http://gerrit.cloudera.org:8080/18226
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
17 files changed, 230 insertions(+), 175 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Zoltan Borok-Nagy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18226
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62
Gerrit-Change-Number: 18226
Gerrit-PatchSet: 10
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>