You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2025/01/15 12:31:37 UTC
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22353
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* file metadata loading needs some ironing
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
38 files changed, 1,619 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13:
(13 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.h
File be/src/service/frontend.h:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.h@93
PS2, Line 93: TGetCatalogInfoResult*
> Pass by reference
We usually pass output parameters by pointers, see above.
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.cc@261
PS2, Line 261:
> Pass by reference
We pass output parameters by pointers.
http://gerrit.cloudera.org:8080/#/c/22353/13/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
http://gerrit.cloudera.org:8080/#/c/22353/13/be/src/service/impala-http-handler.cc@1001
PS13, Line 1001: Value error(status.GetDetail().c_str(), document->GetAllocator());
> nit: this error reporting code piece is used multiple times through this fi
There are indeed lots of identical code pieces, but there are also code pieces that are very similar to this one, but have minor differences. This CR is already quite large, and I rather not introduce any bugs with it.
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/impala-server.cc@3136
PS2, Line 3136: calloncehack::InitializeCallOnceHack();
> Add a TODO and JIRA# to support workload management.
One way to support this is via multi-catalog setups where we have CatalogD.
But I also opened IMPALA-13830 to support workload management in lightweight deployments.
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@54
PS2, Line 54: private static final String KEY_CLIENT_SECRET = "iceberg.rest-catalog.client-secret";
> Does this need to be synchronized?
Done
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@98
PS13, Line 98: DirectMetaProvider
> nit: IcebergMetaProvider
Done
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@128
PS13, Line 128: DirectMetaProvider
> nit: IcebergMetaProvider
Done
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@216
PS13, Line 216: {}.{}
> it should be %s.%s
Good catch, done.
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@190
PS13, Line 190: throw new IllegalStateException(
> IllegalStateException("Create IcebergMetaProvider failed", e)
Done
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@200
PS13, Line 200: Preconditions.checkState(catalogConfigDir != null &&
> It could be checked whether the provided path exists or not
Added the precondition check to listFiles().
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@231
PS13, Line 231: "Expected property %s was not specified.", key));
> Add {configFile} to error messages
Done
http://gerrit.cloudera.org:8080/#/c/22353/13/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@29
PS13, Line 29: import java.net.URLClassLoader;
> nit: unused imports
Done
http://gerrit.cloudera.org:8080/#/c/22353/8/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/8/tests/custom_cluster/test_iceberg_rest_catalog.py@29
PS8, Line 29:
> flake8: E225 missing whitespace around operator
Done
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 Mar 2025 14:09:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 1:
(18 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22353/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22353/1/be/src/common/global-flags.cc@437
PS1, Line 437: DEFINE_string(iceberg_rest_warehouse_location, "", "Iceberg REST catalog warehouse location.");
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@116
PS1, Line 116: Map<String, String> properties) throws IcebergTableLoadingException {
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@103
PS1, Line 103: import static org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_READONLY;
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@488
PS1, Line 488: String partName, Partition msPartition, ListMap<TNetworkAddress> hostIndex)
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@540
PS1, Line 540: ImmutableList<FileDescriptor> fds, ImmutableList<FileDescriptor> insertFds,
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@633
PS1, Line 633: msTable.getPartitionKeysSize(), Lists.newArrayList(msTable.getSd().getLocation()));
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@690
PS1, Line 690: TableMetaRef table, Map<PartitionRef, PartitionMetadata> metas) throws TException {
line too long (133 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@717
PS1, Line 717: TableParams params, Table msTable) throws TException {
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@726
PS1, Line 726: org.apache.iceberg.Table apiTable, ListMap<TNetworkAddress> hostIndex) throws TException {
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@132
PS1, Line 132: public static final String ICEBERG_REST_WAREHOUSE_LOCATION = "iceberg_rest_warehouse_location";
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1070
PS1, Line 1070: private static int createIcebergMetadata(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1115
PS1, Line 1115: private static int createPartitionKeys(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1236
PS1, Line 1236: ICEBERG_REST_URI, ICEBERG_REST_USER_ID, ICEBERG_REST_USER_SECRET, ICEBERG_REST_WAREHOUSE_LOCATION));
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@54
PS1, Line 54: static final String WAREHOUSE_LOCATION = "hdfs://localhost:20500/test-warehouse/iceberg_test/hadoop_catalog";
line too long (111 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@121
PS1, Line 121: // use Map so that Jackson doesn't try to instantiate ImmutableMap from payload.getClass()
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@128
PS1, Line 128: String.format("Failed to serialize and deserialize %s: %s", description, payload), e);
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/1/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/1/tests/custom_cluster/test_iceberg_rest_catalog.py@23
PS1, Line 23: import sys
flake8: F401 'sys' imported but unused
http://gerrit.cloudera.org:8080/#/c/22353/1/tests/custom_cluster/test_iceberg_rest_catalog.py@28
PS1, Line 28: "
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jan 2025 12:33:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11609/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Mar 2025 21:00:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11611/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Mar 2025 09:08:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13: Code-Review+1
(7 comments)
Looks good, I left some nits.
http://gerrit.cloudera.org:8080/#/c/22353/13/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
http://gerrit.cloudera.org:8080/#/c/22353/13/be/src/service/impala-http-handler.cc@1001
PS13, Line 1001: Value error(status.GetDetail().c_str(), document->GetAllocator());
nit: this error reporting code piece is used multiple times through this file, it could be extracted as a function
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@48
PS1, Line 48: IcebergRESTCatalog
> The class name could be IcebergRestCatalog.
This comment lost in PS1 :)
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@98
PS13, Line 98: DirectMetaProvider
nit: IcebergMetaProvider
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@128
PS13, Line 128: DirectMetaProvider
nit: IcebergMetaProvider
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@216
PS13, Line 216: {}.{}
it should be %s.%s
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@200
PS13, Line 200: Preconditions.checkState(catalogConfigDir != null &&
It could be checked whether the provided path exists or not
http://gerrit.cloudera.org:8080/#/c/22353/13/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@29
PS13, Line 29: import java.net.URLClassLoader;
nit: unused imports
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Mar 2025 11:50:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18578/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Mar 2025 21:49:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Mar 2025 17:17:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#7).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,427 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/7
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11570/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 17:17:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#2).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* file metadata loading needs some ironing
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
39 files changed, 1,621 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/2
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17995/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jan 2025 12:57:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#3).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* file metadata loading needs some ironing
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,523 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/3
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 15: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Mar 2025 09:47:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.h
File be/src/service/frontend.h:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.h@93
PS2, Line 93: TGetCatalogInfoResult*
Pass by reference
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/frontend.cc@261
PS2, Line 261: TGetCatalogInfoResult*
Pass by reference
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/service/impala-server.cc@3136
PS2, Line 3136: return Status("Workload management needs CatalogD to be deployed.");
Add a TODO and JIRA# to support workload management.
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@54
PS2, Line 54: public static IcebergRESTCatalog getInstance(TBackendGflags flags) {
Does this need to be synchronized?
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Mar 2025 17:16:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#8).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,447 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/8
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 8:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22353/8/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/8/tests/custom_cluster/test_iceberg_rest_catalog.py@29
PS8, Line 29: =
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 08 Mar 2025 16:24:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 14:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@228
PS14, Line 228: private void checkPropertyValue(String configFile, Properties props, String key, String expectedValue) {
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@231
PS14, Line 231: "Expected property %s was not specified in config file %s.", key, configFile));
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 Mar 2025 14:10:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#15).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dir: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,508 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/15
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18531/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Mar 2025 17:57:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 16: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 16
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Apr 2025 14:36:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18495/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 17:42:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@48
PS1, Line 48: IcebergRESTCatalog
> This comment lost in PS1 :)
Ah, sorry. In the Iceberg library they also use full capital for REST, and in this director we typically just prefix the Iceberg classes with "Iceberg", e.g. HadoopCatalog -> IcebergHadoopCatalog. So I'd lean towards keeping it as is.
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@49
PS1, Line 49: private final static Logger LOG = LoggerFactory.getLogger(IcebergRESTCatalog.class);
> nit: unused
Done
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@116
PS1, Line 116: Map<String, String> properties) throws IcebergTableLoadingException {
> line too long (94 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@228
PS14, Line 228:
> line too long (108 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/14/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@231
PS14, Line 231:
> line too long (91 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Mar 2025 12:17:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Mar 2025 19:46:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#11).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dirs: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,517 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/11
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11581/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 08 Mar 2025 16:22:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11587/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Mar 2025 17:32:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11560/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Mar 2025 14:27:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 4:
(4 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22353/4/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/4/tests/custom_cluster/test_iceberg_rest_catalog.py@25
PS4, Line 25: from tests.common.skip import SkipIfFS
flake8: F401 'tests.common.skip.SkipIfFS' imported but unused
http://gerrit.cloudera.org:8080/#/c/22353/4/tests/custom_cluster/test_iceberg_rest_catalog.py@58
PS4, Line 58: '
flake8: E122 continuation line missing indentation or outdented
http://gerrit.cloudera.org:8080/#/c/22353/4/tests/custom_cluster/test_iceberg_rest_catalog.py@59
PS4, Line 59: s
flake8: E122 continuation line missing indentation or outdented
http://gerrit.cloudera.org:8080/#/c/22353/4/tests/custom_cluster/test_iceberg_rest_catalog.py@60
PS4, Line 60: p
flake8: E122 continuation line missing indentation or outdented
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Mar 2025 16:55:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11327/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jan 2025 18:49:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18518/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 08 Mar 2025 16:48:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18548/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Mar 2025 14:41:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11594/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Mar 2025 15:13:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#9).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
33 files changed, 1,458 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/9
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11610/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Mar 2025 02:51:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11594/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Mar 2025 20:44:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#14).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dirs: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,510 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/14
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18622/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 Mar 2025 14:33:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
(18 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/common/global-flags.cc@437
PS2, Line 437: DEFINE_string(iceberg_rest_warehouse_location, "", "Iceberg REST catalog warehouse location.");
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@116
PS2, Line 116: Map<String, String> properties) throws IcebergTableLoadingException {
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@103
PS2, Line 103: import static org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_READONLY;
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@489
PS2, Line 489: String partName, Partition msPartition, ListMap<TNetworkAddress> hostIndex)
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@541
PS2, Line 541: ImmutableList<FileDescriptor> fds, ImmutableList<FileDescriptor> insertFds,
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@634
PS2, Line 634: msTable.getPartitionKeysSize(), Lists.newArrayList(msTable.getSd().getLocation()));
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@691
PS2, Line 691: TableMetaRef table, Map<PartitionRef, PartitionMetadata> metas) throws TException {
line too long (133 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@718
PS2, Line 718: TableParams params, Table msTable) throws TException {
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@727
PS2, Line 727: org.apache.iceberg.Table apiTable, ListMap<TNetworkAddress> hostIndex) throws TException {
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@132
PS2, Line 132: public static final String ICEBERG_REST_WAREHOUSE_LOCATION = "iceberg_rest_warehouse_location";
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1070
PS2, Line 1070: private static int createIcebergMetadata(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1115
PS2, Line 1115: private static int createPartitionKeys(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1236
PS2, Line 1236: ICEBERG_REST_URI, ICEBERG_REST_USER_ID, ICEBERG_REST_USER_SECRET, ICEBERG_REST_WAREHOUSE_LOCATION));
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@54
PS2, Line 54: static final String WAREHOUSE_LOCATION = "hdfs://localhost:20500/test-warehouse/iceberg_test/hadoop_catalog";
line too long (111 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@121
PS2, Line 121: // use Map so that Jackson doesn't try to instantiate ImmutableMap from payload.getClass()
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@128
PS2, Line 128: String.format("Failed to serialize and deserialize %s: %s", description, payload), e);
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py@23
PS2, Line 23: import sys
flake8: F401 'sys' imported but unused
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py@28
PS2, Line 28: "
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 17:15:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#6).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,430 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/6
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#13).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dirs: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,517 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/13
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@190
PS13, Line 190: throw new IllegalStateException(
IllegalStateException("Create IcebergMetaProvider failed", e)
http://gerrit.cloudera.org:8080/#/c/22353/13/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@231
PS13, Line 231: "Expected property %s was not specified.", key));
Add {configFile} to error messages
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Mar 2025 17:12:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11327/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jan 2025 13:14:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11370/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 22:46:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 3:
PS3 is a rebase
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 14:05:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11514/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 15:22:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18576/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Mar 2025 21:24:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18464/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Mar 2025 17:21:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11570/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 22:50:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11647/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Mar 2025 14:22:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 16:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11685/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 16
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Apr 2025 14:36:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 16: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 16
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Apr 2025 20:04:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/22353/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22353/1//COMMIT_MSG@43
PS1, Line 43: functionlity
typo: functionality
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@48
PS1, Line 48: IcebergRESTCatalog
The class name could be IcebergRestCatalog.
http://gerrit.cloudera.org:8080/#/c/22353/1/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@49
PS1, Line 49: private final static Logger LOG = LoggerFactory.getLogger(IcebergRESTCatalog.class);
nit: unused
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@189
PS2, Line 189: LOG.info(e.toString());
Could be LOG.error or warning
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@231
PS2, Line 231: "invalid state"
It could be something more descriptive
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@277
PS2, Line 277: PartitionRef
This cast is not needed
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@300
PS2, Line 300: TableMetaRefImpl
This cast is not needed
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@315
PS2, Line 315: List<Partition> parts = Collections.emptyList();
As 'parts' is always empty, the for loop above can be removed
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@346
PS2, Line 346: FileMetadataLoader loader) {
nit: formatting
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@597
PS2, Line 597: return false; /* Incremental stats not supported in direct mode */
nit: iceberg?
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@613
PS2, Line 613: DirectMetaProvider
nit: IcebergMetaProvider
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@740
PS2, Line 740: Invalid
It could be more verbose
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@182
PS2, Line 182: return null;
It could use an empty response object instead of null
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1126
PS2, Line 1126: !=
It should be checked with equals() or isEmpty()
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1060
PS2, Line 1060: ContentFile
nit: indentation
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1137
PS2, Line 1137: __HIVE_DEFAULT_PARTITION__
It could be a new constant or DEFAULT_NULL_PARTITION_KEY_VALUE could be reused.
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@54
PS2, Line 54: WAREHOUSE_LOCATION
This location could be calculated from environment variables,
like using DEFAULT_FS
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@66
PS2, Line 66: public void start(boolean join) throws Exception {
This assignment hides the member variable, if 'catalog' is not needed elsewhere, the local scoped variable is enough.
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@115
PS2, Line 115: warn
Could be trace with a conditional before
http://gerrit.cloudera.org:8080/#/c/22353/2/testdata/bin/run-iceberg-rest-server.sh
File testdata/bin/run-iceberg-rest-server.sh:
http://gerrit.cloudera.org:8080/#/c/22353/2/testdata/bin/run-iceberg-rest-server.sh@24
PS2, Line 24: mvn -f "java/iceberg-rest-catalog-test/pom.xml" exec:java \
It could redirect its output to the logs/ folder
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Feb 2025 12:28:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18100/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 17:38:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 14:32:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#4).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,423 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/4
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18472/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Mar 2025 14:57:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11587/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Mar 2025 23:13:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 08 Mar 2025 21:53:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#10).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dirs: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,515 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/10
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11610/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Mar 2025 21:23:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Mar 2025 14:34:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18633/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Mar 2025 12:40:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11370/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 17:13:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11514/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Feb 2025 20:52:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 2:
(35 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22353/2/be/src/common/global-flags.cc@437
PS2, Line 437: DEFINE_string(iceberg_rest_warehouse_location, "", "Iceberg REST catalog warehouse location.");
> line too long (95 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
File fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java@116
PS2, Line 116: Map<String, String> properties) throws IcebergTableLoadingException {
> line too long (94 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@103
PS2, Line 103: import static org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_READONLY;
> line too long (95 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@189
PS2, Line 189: LOG.info(e.toString());
> Could be LOG.error or warning
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@231
PS2, Line 231: "invalid state"
> It could be something more descriptive
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@277
PS2, Line 277: PartitionRef
> This cast is not needed
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@300
PS2, Line 300: TableMetaRefImpl
> This cast is not needed
Most of this method is unneded.
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@315
PS2, Line 315: List<Partition> parts = Collections.emptyList();
> As 'parts' is always empty, the for loop above can be removed
Most of this method is unneded.
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@346
PS2, Line 346: FileMetadataLoader loader) {
> nit: formatting
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@489
PS2, Line 489: String partName, Partition msPartition, ListMap<TNetworkAddress> hostIndex)
> line too long (121 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@541
PS2, Line 541: ImmutableList<FileDescriptor> fds, ImmutableList<FileDescriptor> insertFds,
> line too long (108 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@597
PS2, Line 597: return false; /* Incremental stats not supported in direct mode */
> nit: iceberg?
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@613
PS2, Line 613: DirectMetaProvider
> nit: IcebergMetaProvider
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@634
PS2, Line 634: msTable.getPartitionKeysSize(), Lists.newArrayList(msTable.getSd().getLocation()));
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@691
PS2, Line 691: TableMetaRef table, Map<PartitionRef, PartitionMetadata> metas) throws TException {
> line too long (133 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@718
PS2, Line 718: TableParams params, Table msTable) throws TException {
> line too long (108 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@727
PS2, Line 727: org.apache.iceberg.Table apiTable, ListMap<TNetworkAddress> hostIndex) throws TException {
> line too long (96 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@740
PS2, Line 740: Invalid
> It could be more verbose
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
File fe/src/main/java/org/apache/impala/service/FeCatalogManager.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/FeCatalogManager.java@182
PS2, Line 182: return null;
> It could use an empty response object instead of null
I'm leaning towards null as it expresses better that nothing happened.
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1126
PS2, Line 1126: !=
> It should be checked with equals() or isEmpty()
Added null-check and isEmpty()
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@132
PS2, Line 132: public static final String ICEBERG_REST_WAREHOUSE_LOCATION = "iceberg_rest_warehouse_location";
> line too long (97 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1060
PS2, Line 1060: ContentFile
> nit: indentation
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1070
PS2, Line 1070: private static int createIcebergMetadata(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
> line too long (98 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1115
PS2, Line 1115: private static int createPartitionKeys(org.apache.iceberg.Table iceTbl, FlatBufferBuilder fbb,
> line too long (96 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1137
PS2, Line 1137: __HIVE_DEFAULT_PARTITION__
> It could be a new constant or DEFAULT_NULL_PARTITION_KEY_VALUE could be reu
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1236
PS2, Line 1236: ICEBERG_REST_URI, ICEBERG_REST_USER_ID, ICEBERG_REST_USER_SECRET, ICEBERG_REST_WAREHOUSE_LOCATION));
> line too long (108 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
File java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@54
PS2, Line 54: static final String WAREHOUSE_LOCATION = "hdfs://localhost:20500/test-warehouse/iceberg_test/hadoop_catalog";
> line too long (111 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@54
PS2, Line 54: WAREHOUSE_LOCATION
> This location could be calculated from environment variables,
That's a good idea, I also removed the SkipIf from the python test, so we can try this test on multiple storage systems.
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@66
PS2, Line 66: public void start(boolean join) throws Exception {
> This assignment hides the member variable, if 'catalog' is not needed elsew
Yeah, seems like the member is not needed.
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@115
PS2, Line 115: warn
> Could be trace with a conditional before
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@121
PS2, Line 121: // use Map so that Jackson doesn't try to instantiate ImmutableMap from payload.getClass()
> line too long (99 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java@128
PS2, Line 128: String.format("Failed to serialize and deserialize %s: %s", description, payload), e);
> line too long (97 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/testdata/bin/run-iceberg-rest-server.sh
File testdata/bin/run-iceberg-rest-server.sh:
http://gerrit.cloudera.org:8080/#/c/22353/2/testdata/bin/run-iceberg-rest-server.sh@24
PS2, Line 24: mvn -f "java/iceberg-rest-catalog-test/pom.xml" exec:java \
> It could redirect its output to the logs/ folder
This script can be invoked from the command line for manual testing. But in the python test I switched from PIPE to stdout/stderr.
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py
File tests/custom_cluster/test_iceberg_rest_catalog.py:
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py@23
PS2, Line 23: import sys
> flake8: F401 'sys' imported but unused
Done
http://gerrit.cloudera.org:8080/#/c/22353/2/tests/custom_cluster/test_iceberg_rest_catalog.py@28
PS2, Line 28: "
> flake8: E225 missing whitespace around operator
Done
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Mar 2025 16:54:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#5).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,422 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/5
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11560/
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Mar 2025 20:03:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Apr 2025 14:33:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13586: Initial support for Iceberg REST Catalogs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22353 )
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dir: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Reviewed-on: http://gerrit.cloudera.org:8080/22353
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,508 insertions(+), 28 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 17
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>