You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2023/01/16 17:10:59 UTC

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19423


Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................

IMPALA-11658: Implement Iceberg manifest caching config for Impala

Impala need to supply Iceberg's catalog properties to enable manifest
caching feature. This commit implement the necessary config reading.
Iceberg related config is read from hadoop-conf.xml and supplied as a
Map in catalog instantiation.

Additionally, this patch also replace deprecated RuntimeIOException with
its superclass, UncheckedIOException.

Testing:
- Pass core tests.
- Checked that manifest caching works through debug logging.

Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
---
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopTables.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
7 files changed, 65 insertions(+), 23 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/19423/1
-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12196/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 19:21:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19423

to look at the new patch set (#2).

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................

IMPALA-11658: Implement Iceberg manifest caching config for Impala

Impala needs to supply Iceberg's catalog properties to enable manifest
caching feature. This commit implements the necessary config reading.
Iceberg related config is read from hadoop-conf.xml and supplied as a
Map in catalog instantiation.

Additionally, this patch also replace deprecated RuntimeIOException with
its superclass, UncheckedIOException.

Testing:
- Pass core tests.
- Checked that manifest caching works through debug logging.

Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
---
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopTables.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
7 files changed, 65 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/19423/2
-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12175/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 17:31:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 10:47:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 1: Code-Review+1

(3 comments)

Thanks for working on this, LGTM!

http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@9
PS1, Line 9: need
nit: needs


http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@10
PS1, Line 10: implement
nit: implements


http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@19
PS1, Line 19: - Checked that manifest caching works through debug logging.
Does it works in case of HadoopTables and Catalogs as well?



-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 14:52:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................

IMPALA-11658: Implement Iceberg manifest caching config for Impala

Impala needs to supply Iceberg's catalog properties to enable manifest
caching feature. This commit implements the necessary config reading.
Iceberg related config is read from hadoop-conf.xml and supplied as a
Map in catalog instantiation.

Additionally, this patch also replace deprecated RuntimeIOException with
its superclass, UncheckedIOException.

Testing:
- Pass core tests.
- Checked that manifest caching works through debug logging.

Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Reviewed-on: http://gerrit.cloudera.org:8080/19423
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalogs.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopCatalog.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopTables.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHiveCatalog.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
7 files changed, 65 insertions(+), 23 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 2: Code-Review+2

Thanks for adding this feature!


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 10:47:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19423/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:

http://gerrit.cloudera.org:8080/#/c/19423/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1067
PS1, Line 1067: it is not exist
nit. non-existent or doesn't exist



-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 18:26:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8978/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 10:47:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 16:03:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11658: Implement Iceberg manifest caching config for Impala

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19423 )

Change subject: IMPALA-11658: Implement Iceberg manifest caching config for Impala
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@9
PS1, Line 9: need
> nit: needs
Done


http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@10
PS1, Line 10: implement
> nit: implements
Done


http://gerrit.cloudera.org:8080/#/c/19423/1//COMMIT_MSG@19
PS1, Line 19: - Checked that manifest caching works through debug logging.
> Does it works in case of HadoopTables and Catalogs as well?
I believe manifest caching is only touched if table is loaded using IcebergHadoopCatalog or IcebergHiveCatalog.


http://gerrit.cloudera.org:8080/#/c/19423/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:

http://gerrit.cloudera.org:8080/#/c/19423/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1067
PS1, Line 1067: non-existent.
> nit. non-existent or doesn't exist
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a60a700d2ae6302dfe395d1ef602e6b1d821888
Gerrit-Change-Number: 19423
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jan 2023 19:08:18 +0000
Gerrit-HasComments: Yes