You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "difin (via GitHub)" <gi...@apache.org> on 2026/09/15 20:09:31 UTC
[PR] File planning [hive]
difin opened a new pull request, #6789:
URL: https://github.com/apache/hive/pull/6789
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Hive wires up Iceberg REST server-side scan planning: when `scan-planning-mode=server`, split planning can go through the REST catalog (POST /plan) instead of only the serialized table snapshot on Tez/MR workers. Adds a small `RestCatalogScanPlanning` helper for config and copying catalog settings into job conf, and updates the Iceberg storage handler / input path to reload a live REST table on executors where appropriate. Tests cover configuration, job propagation, and embedded REST planning behavior.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
With a REST catalog that supports `POST /plan`, planning splits on the driver/executor from deserialized `SerializableTable` metadata bypasses the catalog server and can miss server-side optimizations. Operators who set Iceberg’s server planning mode need Hive to honor it end-to-end, including passing catalog connection settings to workers so executors can open a real REST table and delegate `planTasks()` to the server.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducible example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. For REST Iceberg catalogs, operators can set `iceberg.catalog.<catalog>.scan-planning-mode=server` (via `hive-site.xml` or session SET) alongside existing REST catalog properties. When enabled, read jobs that use the Iceberg input format can use server-side scan planning instead of local planning from the serialized snapshot. Default behavior is unchanged when the property is unset or not server. Hive and non-REST catalogs are unaffected.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
`TestRestCatalogScanPlanning` — config keys, server-mode detection, job propagation gating and copying.
`TestRestCatalogScanPlanningServerIT` — embedded REST catalog; Verifies that `RESTTable` / `RESTTableScan` and `PlanTableScanRequest on planTasks()` are used when file planning is on.
`TestHiveIcebergServerSideScanPlanning` — `resolveTableForScanPlanning` when server mode is off or intra-txn metadata is set.
`TestHiveIcebergServerSideScanPlanningServerIT` — Tests that executor job conf with/without propagated catalog props and table reload over REST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135019079
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -343,6 +344,10 @@ public void commitJob(JobContext originalContext) {
public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
setCommonJobConf(jobConf);
configureOutputTableJobConf(tableDesc, jobConf);
+ if (tableDesc != null && tableDesc.getProperties() != null) {
+ String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
Review Comment:
why not move helper into tableDesc? i.e. `tableDesc.getCatalogName()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135107766
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanningServerIT.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
don't we use HiveRESTCatalogServerExtension restCatalogExtension = getHiveRESTCatalogServerExtension() do decorate with additional fixtures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4314127905
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
wonder if we should propose to upstream it in iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135146875
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
Review Comment:
should it be in util package or have util in the name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6230930772
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6521857904
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6030110093
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247571609
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ }
+ return table;
+ }
+
+ private static boolean shouldReloadForServerSideScanPlanning(Configuration conf) {
Review Comment:
Please check the scan variants. Confirm that time travel (SNAPSHOT_ID, as-of timestamp), branch/tag refs, incremental append scans, and metadata tables (tbl.files, tbl.snapshots) either work with RESTTableScan or are explicitly excluded by shouldReload...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247799573
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
Review Comment:
The two `loadTable` paths treat credentials differently: the server-planning branch calls `IcebergVendedCredentialUtil.applyFromJobConf(...)`, the `deserializeTable == null` fallback doesn't, even though both load from the same catalog via `Catalogs.loadTable(conf)`.
Could this be a single load path? E.g. skip deserialization when server-side planning applies and fall through to one `loadTable` + `checkAndSetIoConfig` + `applyFromJobConf` block:
````
Table table = shouldReloadForServerSideScanPlanning(conf) ? null : deserializeTable(conf, tableIdentifier);
if (table == null) {
table = Catalogs.loadTable(conf);
checkAndSetIoConfig(conf, table);
IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
}
return table;
````
Two things to confirm with that shape:
- `applyFromJobConf` is a no-op when the job conf has no vended credentials (the fallback also runs for non-REST catalogs);
- which credentials should win when a fresh REST load already returns vended credentials in `LoadTableResponse` — the fresh ones, or the ones HS2 vended at compile time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164846125
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
Review Comment:
should we call shouldPropagateCatalogPropertiesToJob before propagateCatalogPropertiesToJob and short circuit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300511619
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setCatalogMode(Configuration, String, String)},
+ * {@link #isCatalogServerMode(Configuration, String)}, and
+ * {@link #setHiveMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ private static void setCatalogMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ mode.modeName());
+ }
+
+ public static void setCatalogMode(Configuration conf, String catalogName, String mode) {
+ setCatalogMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ static RESTCatalogProperties.ScanPlanningMode getCatalogMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isCatalogServerMode(Configuration conf, String catalogName) {
+ return getCatalogMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ static boolean isHiveServerMode(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveMode(conf));
+ }
+
+ static String getHiveMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ static boolean isServerSidePlanningEnabled(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerMode(conf) && isCatalogServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return isServerSidePlanningEnabled(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::putIfAbsent);
Review Comment:
Fixed. Propagation now overwrites (`put` instead of `putIfAbsent`) so the session value wins; added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4331622712
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
Done - on the server-side path the ref is now resolved against the serialized table and pinned with `useSnapshot`. One exception: if a branch head was written with an older schema, I keep `useRef`. Iceberg reads branches with the table schema, and `useSnapshot` would switch to the snapshot schema and drop columns added since. Added ITs for both cases: a branch advanced after compile, and a branch with an older schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4189601816
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
Review Comment:
Removed the `catalogPropertyKey()` wrapper; `setScanPlanningMode` / `getScanPlanningMode` now use `IcebergCatalogProperties.catalogPropertyConfigKey(..., RESTCatalogProperties.SCAN_PLANNING_MODE)` like the rest of iceberg-handler. Tests updated accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201563695
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4425480868
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,27 +162,45 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table serializedTable = HiveTableUtil.deserializeTable(conf, tableIdentifier);
+
final ExecutorService workerPool =
ThreadPools.newFixedThreadPool("iceberg-plan-worker-pool",
conf.getInt(SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(), ThreadPools.WORKER_THREAD_POOL_SIZE));
try {
- return planInputSplits(table, conf, workerPool);
+ if (HiveTableUtil.shouldReloadForServerSideScanPlanning(conf, serializedTable)) {
+ String catalogName = HiveTableUtil.resolveCatalogName(conf);
+ try (CloseableRestCatalog catalog = CloseableRestCatalog.load(conf, catalogName)) {
+ Table planningTable = catalog.loadTable(tableIdentifier);
+ HiveTableUtil.checkAndSetIoConfig(conf, planningTable);
+ return planInputSplits(planningTable, conf, workerPool, tableIdentifier, serializedTable);
Review Comment:
THX!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4375526935
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
`overlayTableProperties` already loads the table (query-cached via `IcebergTableUtil.getTable`) and sets the marker from `usesServerSideScanPlanning(table)`, and `propagateVendedCredentialsToJob` loads it again from the same cache. Could `configureInputJobCredentials` take it once and gate the catalog secrets on the same decision as the marker? Right now the secrets ship for every REST table with the kill switch on, including ones Iceberg resolved to client-side planning:
````
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
Properties props = tableDesc.getProperties();
boolean vended = IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf);
if (!vended && !RestCatalogScanPlanningUtil.isServerSideScanPlanningEnabled(conf)) {
return;
}
Table table;
try {
table = vended ?
IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, conf) :
IcebergTableUtil.getTable(conf, props);
} catch (NoSuchTableException e) {
return; // CTAS: table doesn't exist yet
}
RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(table, tableDesc, secrets, conf); // gated on usesServerSideScanPlanning(table)
if (vended) {
IcebergVendedCredentialUtil.propagateVendedCredentialsToJob(table, tableDesc, secrets, conf);
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub) via gitbox" <gi...@hive.apache.org>.
difin merged PR #6789:
URL: https://github.com/apache/hive/pull/6789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135126023
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanningServerIT.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
is this a test framefork class or a test class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135177468
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
Review Comment:
i see you use mix of static imports, catalogPropertyKey in `setScanPlanningMode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135255082
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
Review Comment:
you check tableIdentifier and set executorTableId, that is inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164828587
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
Review Comment:
is there still MR job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247428118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
setCatalogMode ? or setScanPlanningMode ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247428118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
could we name these after the concept they control, e.g. `setScanPlanningMode` / `getScanPlanningMode` / `isServerSideScanPlanning`?
`setCatalogMode` reads like it sets some general catalog mode, and today we have three predicates (`isCatalogServerMode`, `isHiveServerMode`, `isServerSidePlanningEnabled`). This also lines up with Iceberg's `RESTCatalogProperties.ScanPlanningMode`.
see https://github.com/apache/hive/pull/6789#discussion_r4247714315 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247899802
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setCatalogMode(Configuration, String, String)},
+ * {@link #isCatalogServerMode(Configuration, String)}, and
+ * {@link #setHiveMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ private static void setCatalogMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ mode.modeName());
+ }
+
+ public static void setCatalogMode(Configuration conf, String catalogName, String mode) {
+ setCatalogMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ static RESTCatalogProperties.ScanPlanningMode getCatalogMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isCatalogServerMode(Configuration conf, String catalogName) {
+ return getCatalogMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ static boolean isHiveServerMode(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveMode(conf));
+ }
+
+ static String getHiveMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ static boolean isServerSidePlanningEnabled(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerMode(conf) && isCatalogServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return isServerSidePlanningEnabled(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::putIfAbsent);
Review Comment:
propagation never overwrites existing keys, so a session SET can’t replace an older value already in job properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4400206533
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,27 +162,45 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table serializedTable = HiveTableUtil.deserializeTable(conf, tableIdentifier);
+
final ExecutorService workerPool =
ThreadPools.newFixedThreadPool("iceberg-plan-worker-pool",
conf.getInt(SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(), ThreadPools.WORKER_THREAD_POOL_SIZE));
try {
- return planInputSplits(table, conf, workerPool);
+ if (HiveTableUtil.shouldReloadForServerSideScanPlanning(conf, serializedTable)) {
+ String catalogName = HiveTableUtil.resolveCatalogName(conf);
+ try (CloseableRestCatalog catalog = CloseableRestCatalog.load(conf, catalogName)) {
+ Table planningTable = catalog.loadTable(tableIdentifier);
+ HiveTableUtil.checkAndSetIoConfig(conf, planningTable);
+ return planInputSplits(planningTable, conf, workerPool, tableIdentifier, serializedTable);
Review Comment:
@difin , please open a JIRA and link it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201643192
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,13 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SCAN_PLANNING_MODE(
+ "hive.iceberg.rest.scan-planning-mode", "client",
+ new StringSet("client", "server"),
+ "Hive-wide default for Iceberg REST catalog scan planning in Tez/MR split generation, aligned with\n" +
+ "catalog property scan-planning-mode. In server mode, catalog settings are propagated to jobs and\n" +
Review Comment:
Simplified and removed unnecessary implementation details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202300333
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
Reload here means `Catalogs.loadTable` instead of the serialized `SerializableTable`, so we get a live `RESTTable` for POST /plan, only in `getSplits` when REST + server scan mode are on (skipped for intra-txn TABLE_METADATA_LOCATION). It’s extra REST metadata cost vs deserialize, but it runs at split-planning time, not per task/row; multiple scans of the same table in one query can each hit `loadTable` once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4034298144
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
Review Comment:
I believe this will create a new RestCatalog Object/HttpClient on every call, curious if this can cause any connection leak?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5840476577
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4340650201
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
Done. The marker is now set in `overlayTableProperties` from the table Iceberg loaded (a `RESTTable`, checked by class name, gated by the kill switch), so a server-required mode is followed even when the client property is unset. It's keyed per table (`iceberg.mr.server-side-scan-planning.<table>`) even though it's in per-table job properties, because `HiveInputFormat.getSplits` and `FetchOperator` reuse one conf across tables, so an unkeyed flag would leak to the next table. configureJobConf no longer writes the marker. Job secrets now ship catalog properties for any REST catalog with the kill switch on, since the server decides the mode per table. Added ITs for a server-decided mode with no client property, and for the kill switch overriding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6210094497
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300410648
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
Review Comment:
Unified credential setup: Both catalog loads now go through `configureLoadedTable` (IO config + `applyFromJobConf`).
Kept them as separate branches in `getSplits` because the server path has to keep the serialized table for snapshot pinning and close the catalog after planning.
- `applyFromJobConf` is a no-op without serialized creds or the delegation header
- credentials from the fresh `LoadTableResponse` win over the ones vended at compile time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5702666506
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4190197118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!RestCatalogScanPlanning.shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
Review Comment:
there were a few methods that should stay public to be visible for testing from another package (@VisibleForTesting annotation is not suitable here because the style check requires visibility to be package-private on methods annotated with VisibleForTesting), and few other public methods whose visibility could be reduced - done.
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!RestCatalogScanPlanning.shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
Review Comment:
there were a few methods that should stay public to be visible for testing from another package (VisibleForTesting annotation is not suitable here because the style check requires visibility to be package-private on methods annotated with VisibleForTesting), and few other public methods whose visibility could be reduced - done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164893701
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
+ conf.set(InputFormatConfig.TABLE_IDENTIFIER, executorTableId);
+ }
+ if (conf.get(InputFormatConfig.SERIALIZED_TABLE_PREFIX + executorTableId) == null) {
Review Comment:
can we extract to local var not to repeat in set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
Please add test validating that credential / token keys stay out of the propagated job properties.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM (see Tanishq [comment](https://github.com/apache/hive/pull/6789#discussion_r4034298144)).
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247563182
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setCatalogMode(Configuration, String, String)},
+ * {@link #isCatalogServerMode(Configuration, String)}, and
+ * {@link #setHiveMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ private static void setCatalogMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ mode.modeName());
+ }
+
+ public static void setCatalogMode(Configuration conf, String catalogName, String mode) {
+ setCatalogMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ static RESTCatalogProperties.ScanPlanningMode getCatalogMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isCatalogServerMode(Configuration conf, String catalogName) {
Review Comment:
Server mode needs both hive.iceberg.rest.scan-planning-mode=server and the per-catalog scan-planning-mode=server
The REST server can advertise the planning mode in LoadTableResponse.config, which is exactly what the IntegrationTests simulate, so the Hive-side check can disagree with what the loaded table will actually do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135266247
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
Review Comment:
maybe simply tableId ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5737809057
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [6 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4340703299
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -278,17 +278,8 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
- if (!IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
- return;
- }
- try {
- Table table =
- IcebergVendedCredentialUtil.getTableWithVendedCredentials(tableDesc.getProperties(), conf);
- String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
- IcebergVendedCredentialUtil.propagateToJob(table, catalogName, null, secrets, conf);
- } catch (NoSuchTableException ex) {
- // Table may not exist yet for CTAS; credentials will not be available.
- }
+ RestCatalogScanPlanningUtil.propagateInputJobCredentials(tableDesc, secrets, conf);
Review Comment:
Done, renamed to `RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob` and `IcebergVendedCredentialUtil.propagateVendedCredentialsToJob`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4314127905
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
Instead of fixing `close()` here, could we drop `CloseableRestCatalog` and use Iceberg's `org.apache.iceberg.io.CloseableGroup` in `getSplits`? I
t closes any `AutoCloseable` (`RESTCatalog` is `Closeable`) and with `setSuppressCloseFailure(true)` logs close failures instead of throwing:
````
try (CloseableGroup closer = new CloseableGroup()) {
closer.setSuppressCloseFailure(true);
if (catalog instanceof AutoCloseable closeable) {
closer.addCloseable(closeable);
}
...
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313671696
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
On `isSecretCatalogPropertySuffix`: a substring denylist both under-matches (`header.Authorization` / `header.X-Api-Key`, custom `AuthManager` properties) and over-matches (`oauth2-server-uri`, `token-refresh-enabled`).
Since only the AM's `loadCatalog` needs these properties (executors read through the serialized table, and the commit runs in HS2 on the session conf) and job secrets already reach split generation via `addSplitsForGroup`, could we route all `iceberg.catalog.<name>.*` keys through job secrets and keep only the marker and `metastore.catalog.default` in plain job properties?
That removes the classification entirely. Note that `configureJobConf` currently writes the catalog keys directly into the JobConf, so it would need to set only the marker as well.
If you'd rather keep some keys visible, please flip to an allowlist of known non-sensitive keys (type, uri, warehouse, catalog-impl, io-impl, prefix, scan-planning-mode) so unknown keys default to secrets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313671696
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
On `isSecretCatalogPropertySuffix`: a substring denylist both under-matches (`header.Authorization` / `header.X-Api-Key`, custom `AuthManager` properties) and over-matches (`oauth2-server-uri`, `token-refresh-enabled`).
Since only the AM's `loadCatalog` needs these properties (executors read through the serialized table) and job secrets already reach split generation via `addSplitsForGroup`, could we route all `iceberg.catalog.<name>.*` keys through job secrets and keep only the marker and `metastore.catalog.default` in plain job properties?
That removes the classification entirely. If you'd rather keep some keys visible, please flip to an allowlist of known non-sensitive keys (type, uri, warehouse, catalog-impl, io-impl, prefix, scan-planning-mode) so unknown keys default to secrets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4418608658
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,27 +162,45 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table serializedTable = HiveTableUtil.deserializeTable(conf, tableIdentifier);
+
final ExecutorService workerPool =
ThreadPools.newFixedThreadPool("iceberg-plan-worker-pool",
conf.getInt(SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(), ThreadPools.WORKER_THREAD_POOL_SIZE));
try {
- return planInputSplits(table, conf, workerPool);
+ if (HiveTableUtil.shouldReloadForServerSideScanPlanning(conf, serializedTable)) {
+ String catalogName = HiveTableUtil.resolveCatalogName(conf);
+ try (CloseableRestCatalog catalog = CloseableRestCatalog.load(conf, catalogName)) {
+ Table planningTable = catalog.loadTable(tableIdentifier);
+ HiveTableUtil.checkAndSetIoConfig(conf, planningTable);
+ return planInputSplits(planningTable, conf, workerPool, tableIdentifier, serializedTable);
Review Comment:
Opened [HIVE-30230](https://issues.apache.org/jira/browse/HIVE-30230) and linked it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6510885761
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub) via gitbox" <gi...@hive.apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6603875598
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164754484
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,13 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SCAN_PLANNING_MODE(
+ "hive.iceberg.rest.scan-planning-mode", "client",
+ new StringSet("client", "server"),
+ "Hive-wide default for Iceberg REST catalog scan planning in Tez/MR split generation, aligned with\n" +
Review Comment:
there is no MR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135194388
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
Review Comment:
this seem to be a repeated functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135183484
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
naming seems to repeat class name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135293063
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
what is this reload, isn't it expensive? what if we have multiple requests in same query/session?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247674427
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
`hive.iceberg.rest.server-side-scan-planning.enabled` = true | false.
The helper methods then each answer one question:
> getScanPlanningMode(conf, catalogName) → ScanPlanningMode
What does the catalog want?
> isServerSideScanPlanningEnabled(conf)
Has Hive disabled the feature?
> shouldUseServerSideScanPlanning(conf, catalogName)
Mode is SERVER and the switch is on
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
`hive.iceberg.rest.server-side-scan-planning.enabled` = true | false.
The helper methods then each answer one question:
> getScanPlanningMode(conf, catalogName) → ScanPlanningMode
What does the catalog want?
> isServerSideScanPlanningEnabled(conf)
Has Hive disabled the feature?
> shouldUseServerSideScanPlanning(conf, catalogName)
Mode is SERVER and the switch is on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
Also a self-join or union over the same table can read two different snapshots in one query.
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
please add test for the drift case: serialize the table, commit a new snapshot through the catalog, then call `resolveTableForScanPlanning` / `getSplits` and assert the planned tasks belong to the serialized snapshot, not the new one
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
Could we either restrict propagation to an allowlist of non-secret connection keys (type, uri, warehouse, scan-planning-mode, ...) or route `credential` / `token` / OAuth settings through the job secrets path, the same way `configureInputJobCredentials` handles vended credentials?
Please add test validating that credential / token keys stay out of the propagated job properties.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM (see Tanishq [comment](https://github.com/apache/hive/pull/6789#discussion_r4034298144)).
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247674427
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
- `hive.iceberg.rest.server-side-scan-planning.enabled` (boolean)
- the mode itself comes only from the catalog / what Iceberg resolves on the loaded table
Helpers then become:
- `getScanPlanningMode(conf, catalogName)` → `ScanPlanningMode`
- `isServerSideScanPlanningEnabled(conf)` → the kill switch
- `shouldUseServerSideScanPlanning(conf, catalogName)` → mode is `SERVER` and the switch is on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201529654
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
Review Comment:
It checks if `conf.get(InputFormatConfig.TABLE_IDENTIFIER) == null` then it sets `table.name()` as the table identifier. Do you mean the name is inconsistent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300478644
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ }
+ return table;
+ }
+
+ private static boolean shouldReloadForServerSideScanPlanning(Configuration conf) {
Review Comment:
Done: HS2 sets iceberg.mr.server-side-scan-planning.<catalog>=true when it decides on server-side planning; getSplits only checks that marker (plus the incremental / metadata-table / intra-txn exclusions) and no longer re-derives type/mode/switch. Keyed per catalog because one job conf can carry tables from several catalogs. If the marker is set and the reload fails, the error propagates instead of falling back silently. The pinned snapshot comes from the serialized table, so no separate snapshot-id key is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4418479909
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -279,8 +279,23 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
- RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(tableDesc, secrets, conf);
- IcebergVendedCredentialUtil.propagateVendedCredentialsToJob(tableDesc, secrets, conf);
+ Properties props = tableDesc.getProperties();
+ if (props == null) {
+ return;
+ }
+ if (!IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf) &&
+ !RestCatalogScanPlanningUtil.isServerSideScanPlanningEnabled(conf)) {
+ return;
+ }
+ Table table;
+ try {
+ table = IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, conf);
+ } catch (NoSuchTableException e) {
+ // Table may not exist yet for CTAS; there is nothing to plan or vend credentials for.
+ return;
+ }
+ RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(table, tableDesc, secrets, conf);
Review Comment:
Done, as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6348220830
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [32 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4340926786
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
Done, dropped `CloseableRestCatalog. getSplits` now registers the catalog with a `CloseableGroup` when it's `AutoCloseable` (`RESTCatalog` is `Closeable`), with `setSuppressCloseFailure(true)`, so a close failure is logged instead of failing split generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4375394603
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)},
+ * {@link #getScanPlanningMode(Configuration, String)} and
+ * {@link #propagateCatalogSecretsToJob(TableDesc, Map, Configuration)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private static final String REST_TABLE_CLASS_NAME = "org.apache.iceberg.rest.RESTTable";
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when Iceberg resolved the loaded table for server-side scan planning. Iceberg decides
+ * this per table, letting the mode in the server's {@code LoadTableResponse} take precedence over the
+ * client {@code scan-planning-mode}, and then returns a package-private {@code RESTTable}.
+ */
+ static boolean usesServerSideScanPlanning(Table table) {
+ return table != null && REST_TABLE_CLASS_NAME.equals(table.getClass().getName());
Review Comment:
why not use RESTTable.class.getName() directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4190197118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!RestCatalogScanPlanning.shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
Review Comment:
there were a few methods that need to be visible-for-testing (added the annotation in code), and few other public methods whose visibility could be reduced - done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202038118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> {
+ if (jobConf.get(key) == null) {
+ jobConf.set(key, value);
+ }
+ });
+ }
+
+ private static void propagateCatalogProperties(
+ Configuration sessionConf, String catalogName, PropertyConsumer consumer) {
+ String resolvedCatalogName = resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(resolvedCatalogName, sessionConf)) {
+ return;
+ }
+
+ consumer.accept(
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE.varname,
+ getHiveRestScanPlanningMode(sessionConf));
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ consumer.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (entry.getKey().startsWith(catalogPrefix)) {
+ consumer.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ private interface PropertyConsumer {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4190357115
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanning.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.nio.file.Path;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Unit tests for Hive server-side REST catalog scan planning via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}. Embedded REST server coverage is in
+ * {@code TestHiveIcebergServerSideScanPlanningServerIT} in {@code itests/hive-iceberg-rest-server}.
+ */
+class TestHiveIcebergServerSideScanPlanning {
Review Comment:
You’re right — `usesDeserializedTable* `tests in `TestHiveIcebergServerSideScanPlanning` only check that we don’t reload from the catalog, not the positive server-side path. This class-level javadoc was leftover from when this was a single test class; I later split it into `TestHiveIcebergServerSideScanPlanning` (unit, reload guards) and `TestHiveIcebergServerSideScanPlanningServerIT` (embedded REST server, propagation + reload). I’ve moved the test class description to the IT test class and tightened the unit-test's class javadoc. Sorry for the confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201881737
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
Review Comment:
Thanks for catching the typo, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201893880
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
Review Comment:
Renamed both the call site's method and the method it calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201872861
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
Review Comment:
renamed to `isServerSidePlanningEnabled` - meaning both catalog properties include server mode and Hive feature flag is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202109695
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
Review Comment:
Addressed the duplication with a single private path and thin Map / JobConf overloads (BiConsumer, same idea as the two `isV2TableOrAbove` overloads). `BinaryOperator` doesn’t seem to fit here: in `isV2TableOrAbove` it’s get-with-default; here we need put-if-absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202143999
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
+ conf.set(InputFormatConfig.TABLE_IDENTIFIER, executorTableId);
+ }
+ if (conf.get(InputFormatConfig.SERIALIZED_TABLE_PREFIX + executorTableId) == null) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add REST catalog server-side scan planning support [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5698590495
> There were already discussions happening under [HIVE-29599](https://issues.apache.org/jira/browse/HIVE-29599) for server side planning?
This is different - this PR is about adding client-side support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247361609
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
Review Comment:
fixed now. in the if you checked `tableIdentifier == null` but it set `executorTableId` was used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247428118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
setCatalogMode ? or set/getScanPlanningMode, isServerSideScanPlanningMode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164872860
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> {
+ if (jobConf.get(key) == null) {
+ jobConf.set(key, value);
+ }
+ });
+ }
+
+ private static void propagateCatalogProperties(
+ Configuration sessionConf, String catalogName, PropertyConsumer consumer) {
+ String resolvedCatalogName = resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(resolvedCatalogName, sessionConf)) {
+ return;
+ }
+
+ consumer.accept(
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE.varname,
+ getHiveRestScanPlanningMode(sessionConf));
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ consumer.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (entry.getKey().startsWith(catalogPrefix)) {
+ consumer.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ private interface PropertyConsumer {
Review Comment:
why invent the wheel, can't we use BiConsumer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313588904
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
+
+ private final Catalog catalog;
+
+ public static CloseableRestCatalog load(Configuration conf, String catalogName) {
+ return Catalogs.loadCatalog(conf, catalogName)
+ .map(CloseableRestCatalog::new)
+ .orElseThrow(
+ () ->
+ new IllegalStateException(
+ "Iceberg catalog is required for REST server-side scan planning"));
+ }
+
+ private CloseableRestCatalog(Catalog catalog) {
+ this.catalog = catalog;
+ }
+
+ public Table loadTable(String tableIdentifier) {
+ return catalog.loadTable(TableIdentifier.parse(tableIdentifier));
+ }
+
+ @Override
+ public void close() {
+ if (catalog instanceof AutoCloseable closeable) {
+ try {
+ closeable.close();
+ } catch (Exception e) {
+ throw new UncheckedIOException(
Review Comment:
should we throw exception or simply LOG.warn("Failed to close Iceberg catalog {}", catalog.name(), e);?
close failure kills a query that already planned successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313671696
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
should we catch `header.*` (e.g. `header.Authorization`) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4392050350
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)},
+ * {@link #getScanPlanningMode(Configuration, String)} and
+ * {@link #propagateCatalogSecretsToJob(TableDesc, Map, Configuration)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private static final String REST_TABLE_CLASS_NAME = "org.apache.iceberg.rest.RESTTable";
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when Iceberg resolved the loaded table for server-side scan planning. Iceberg decides
+ * this per table, letting the mode in the server's {@code LoadTableResponse} take precedence over the
+ * client {@code scan-planning-mode}, and then returns a package-private {@code RESTTable}.
+ */
+ static boolean usesServerSideScanPlanning(Table table) {
+ return table != null && REST_TABLE_CLASS_NAME.equals(table.getClass().getName());
Review Comment:
This doesn't work because `RESTTable` is package-private in Iceberg 1.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4400138483
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -279,8 +279,23 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
- RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(tableDesc, secrets, conf);
- IcebergVendedCredentialUtil.propagateVendedCredentialsToJob(tableDesc, secrets, conf);
+ Properties props = tableDesc.getProperties();
+ if (props == null) {
+ return;
+ }
+ if (!IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf) &&
+ !RestCatalogScanPlanningUtil.isServerSideScanPlanningEnabled(conf)) {
+ return;
+ }
+ Table table;
+ try {
+ table = IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, conf);
+ } catch (NoSuchTableException e) {
+ // Table may not exist yet for CTAS; there is nothing to plan or vend credentials for.
+ return;
+ }
+ RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(table, tableDesc, secrets, conf);
Review Comment:
Could we simplify how credentials reach the job? Right now `configureInputJobCredentials` passes `table` + `tableDesc` into two helpers that each re-check null props, re-evaluate their own gate, and read `CATALOG_NAME` (one with the session-default fallback, one raw). Underneath, `propagateToJob` takes two out-param maps and callers pass `null` to select a half.
Proposal:
**1. Return both halves instead of writing into nullable maps**
```java
/** Vended credentials split per HIVE-20651: plain job properties vs job secrets. */
record VendedJobConf(Map<String, String> properties, Map<String, String> secrets) {
static final VendedJobConf EMPTY = new VendedJobConf(Map.of(), Map.of());
}
static VendedJobConf toJobConf(Table table, String catalogName, Configuration conf) {
List<StorageCredential> credentials =
withConfigurationOverrides(catalogName, extractCredentials(table), conf);
if (credentials.isEmpty()) {
return VendedJobConf.EMPTY;
}
Map<String, String> properties = new LinkedHashMap<>();
Map<String, String> secrets = new LinkedHashMap<>();
secrets.put(
InputFormatConfig.vendedCredentialsKey(table.name()),
serializeToSingleLineBase64(Lists.newArrayList(credentials)));
for (StorageCredential credential : credentials) {
addCredentialEntries(catalogName, credential, properties, secrets, conf);
}
return new VendedJobConf(properties, secrets);
}
```
The `!= null` checks in `addCredentialEntries` / `addCredentialEntry` go away, and `propagateToJob` / `propagateVendedCredentialsToJob` can be dropped.
**2. Let `configureInputJobCredentials` own the decisions**
```java
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
Properties props = tableDesc.getProperties();
if (props == null) {
return;
}
boolean vendedCredentials = IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf);
if (!vendedCredentials && !RestCatalogScanPlanningUtil.isServerSideScanPlanningEnabled(conf)) {
return;
}
Table table;
try {
table = IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, conf);
} catch (NoSuchTableException e) {
// Table may not exist yet for CTAS; there is nothing to plan or vend credentials for.
return;
}
String catalogName = HiveTableUtil.resolveCatalogName(conf, props.getProperty(InputFormatConfig.CATALOG_NAME));
if (RestCatalogScanPlanningUtil.shouldPlanServerSide(conf, table)) {
RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(catalogName, secrets, conf);
}
if (vendedCredentials) {
secrets.putAll(IcebergVendedCredentialUtil.toJobConf(table, catalogName, conf).secrets());
}
}
```
with `propagateCatalogSecretsToJob` reduced to the copy itself:
```java
public static void propagateCatalogSecretsToJob(
String catalogName, Map<String, String> jobSecrets, Configuration sessionConf) {
if (StringUtils.isEmpty(catalogName)) {
return;
}
String catalogPrefix = IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + catalogName + ".";
sessionConf.forEach(entry -> {
if (entry.getKey().startsWith(catalogPrefix)) {
jobSecrets.put(entry.getKey(), entry.getValue());
}
});
}
```
**3. Other call sites**
```java
// configureInputJobProperties
if (isVendedCredentials) {
IcebergVendedCredentialUtil.toJobConf(table, catalogName, configuration)
.properties().forEach(map::putIfAbsent);
table = IcebergVendedCredentialUtil.secretFreeCopy(table, configuration);
}
// refreshVendedCredentialsIfMissing
VendedJobConf vended = toJobConf(table, catalogName, configuration);
vended.properties().forEach(jobConf::set);
mergeJobSecrets(tableDesc, vended.secrets());
```
Here, `catalogName` should be resolved the same way in all three places (`HiveTableUtil.resolveCatalogName`). Today the secrets path falls back to the session default, the vended paths use the raw `CATALOG_NAME` (null for default-catalog tables, so `withConfigurationOverrides` skips session overrides), and `refreshVendedCredentialsIfMissing` returns early on null. Is that difference intended?
Tests: `TestRestCatalogScanPlanningUtil.propagateCatalogSecretsToJobSkipsTableNotPlannedServerSide` would move to the storage handler (the gate now lives there), and the IT call at `TestHiveIcebergServerSideScanPlanningServerIT:404` needs the new signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5822111858
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247428118
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
could we name these after the concept they control, e.g. `setScanPlanningMode` / `getScanPlanningMode` / `isServerSideScanPlanning`?
`setCatalogMode` reads like it sets some general catalog mode, and today we have three predicates (`isCatalogServerMode`, `isHiveServerMode`, `isServerSidePlanningEnabled`). This also lines up with Iceberg's `RESTCatalogProperties.ScanPlanningMode`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164836393
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
Review Comment:
weird. method name is validate (shouldPropagateCatalogPropertiesToJob) but it calls requestsServerSidePlanning? could we improve/refactor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6044707329
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4189929139
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
Renamed to drop the redundant 'ScanPlanning' in the method names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202032931
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4332041588
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
Done, went with the first option. All `iceberg.catalog.<name>.*` keys now go through job secrets. Job properties and configureJobConf set only the marker and `metastore.catalog.default`, so the denylist is gone. Unit tests now cover custom keys like `header.Authorization` going to secrets, and the IT feeds the AM conf from secrets as `addSplitsForGroup` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313671696
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
should we catch `header.*` (e.g. iceberg.catalog.ice.header.Authorization=Bearer … or header.X-Api-Key) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300511619
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setCatalogMode(Configuration, String, String)},
+ * {@link #isCatalogServerMode(Configuration, String)}, and
+ * {@link #setHiveMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ private static void setCatalogMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ mode.modeName());
+ }
+
+ public static void setCatalogMode(Configuration conf, String catalogName, String mode) {
+ setCatalogMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ static RESTCatalogProperties.ScanPlanningMode getCatalogMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isCatalogServerMode(Configuration conf, String catalogName) {
+ return getCatalogMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ static boolean isHiveServerMode(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveMode(conf));
+ }
+
+ static String getHiveMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ static boolean isServerSidePlanningEnabled(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerMode(conf) && isCatalogServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return isServerSidePlanningEnabled(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::putIfAbsent);
Review Comment:
Fixed. Propagation now overwrites (put / set) so the session value wins; added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313652650
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
For `t.branch_x`, `OUTPUT_TABLE_SNAPSHOT_REF` is set, `hasExplicitScanSnapshotBinding` returns true, and `scan.useRef(...)` resolves the branch on the live `RESTTable` in the AM. Could we resolve that ref against the serialized table too (as done for `SNAPSHOT_ID` names) and add an IT that advances the branch after compile?
````
if (StringUtils.isNotEmpty(refName)) {
String branch = HiveUtils.getTableSnapshotRef(refName);
if (serverSidePinTable != null) {
SnapshotRef ref = serverSidePinTable.refs().get(branch);
if (ref == null) {
throw new RuntimeException("Cannot find matching reference name " + branch);
}
scan = scan.useSnapshot(ref.snapshotId()); // compile-time branch head
} else {
scan = scan.useRef(branch); // client-side path unchanged
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313885820
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
The server-vs-client decision is made at HS2 from the client-side `iceberg.catalog.<name>.scan-planning-mode` only (`getScanPlanningMode`), but Iceberg itself (`RESTSessionCatalog.restTableForScanPlanning`) lets the mode returned in `LoadTableResponse` take precedence over the client property. So if the server requires server-side planning and the client property is unset, Iceberg returns a `RESTTable` while Hive skips the marker and plans locally from manifests.
`overlayTableProperties` already loads the table from the catalog; could the marker be set based on what Iceberg resolved there (the loaded table being a server-planning `RESTTable`, gated by the kill switch) instead of re-reading the client property? Note:
- the server's mode is per table, so the marker would need to be keyed per table rather than per catalog;
- `RESTTable` is package-private, so this probably needs a small helper (e.g. in patched-iceberg-core) or a class-name check for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4383647312
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
Done. `configureInputJobCredentials` now loads the table once (from the query cache via `getTableWithVendedCredentials`), and catalog secrets are gated on the same `shouldPlanServerSide(conf, table)` check as the marker, so REST tables that Iceberg resolved to client-side planning no longer get them. I removed the old catalog-type gate, `propagateVendedCredentialsToJob` keeps its own vended check, and I added unit and IT coverage for both the positive and negative cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6457720822
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4389214998
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
did you push?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub) via gitbox" <gi...@hive.apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6616724306
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4395437098
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
Yes, I pushed today.
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
Yes, I pushed today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5786527682
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4051175790
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
Review Comment:
Yes - `Catalogs.loadTable` creates a new REST catalog (and HTTP client) on each call, and we don’t close it on this path today. That’s the same pattern as other `Catalogs.loadTable` usages in the handler. Here it runs once per `getSplits` during split planning, not per input split. A reload only happens when `shouldReloadForServerSideScanPlanning` is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4289774107
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
Done — introduced the Hive-side kill switch `hive.iceberg.rest.server-side-scan-planning.enabled` (boolean); the mode comes only from the catalog's scan-planning-mode. Helpers are now `getScanPlanningMode` / `setScanPlanningMode`, `isServerSideScanPlanningEnabled` and `shouldUseServerSideScanPlanning` replacing the three old predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164768940
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,13 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SCAN_PLANNING_MODE(
+ "hive.iceberg.rest.scan-planning-mode", "client",
+ new StringSet("client", "server"),
+ "Hive-wide default for Iceberg REST catalog scan planning in Tez/MR split generation, aligned with\n" +
+ "catalog property scan-planning-mode. In server mode, catalog settings are propagated to jobs and\n" +
Review Comment:
why do we need these details here? this is user facing config
````
In server mode, catalog settings are propagated to jobs and\n" +
"executors reload a REST table so planning can use POST /plan. In client mode (default), split\n" +
"generation uses the serialized table snapshot even if a catalog requests server-side scan planning.
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164863173
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> {
+ if (jobConf.get(key) == null) {
+ jobConf.set(key, value);
+ }
+ });
+ }
+
+ private static void propagateCatalogProperties(
+ Configuration sessionConf, String catalogName, PropertyConsumer consumer) {
+ String resolvedCatalogName = resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(resolvedCatalogName, sessionConf)) {
Review Comment:
again repeated check. why is it inside action
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164825082
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
Review Comment:
requestsServerSideScanPlanning ? idk or simplify other namings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201576459
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,13 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SCAN_PLANNING_MODE(
+ "hive.iceberg.rest.scan-planning-mode", "client",
+ new StringSet("client", "server"),
+ "Hive-wide default for Iceberg REST catalog scan planning in Tez/MR split generation, aligned with\n" +
Review Comment:
Removed"MR"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201797721
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
Review Comment:
This has been addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135093112
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestRestCatalogScanPlanningServerIT.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.rest.requests.PlanTableScanRequest;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/**
+ * Embedded REST server tests for {@link RestCatalogScanPlanning}: when Hive-style configuration sets
+ * {@code scan-planning-mode=server}, {@link RESTTable} / {@link RESTTableScan} delegate split
+ * planning to the catalog server ({@code POST /plan}).
+ */
+class TestRestCatalogScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
don't we use HiveRESTCatalogServerExtension restCatalogExtension = getHiveRESTCatalogServerExtension() do decorate with additional fixtures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4134983810
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,12 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED(
Review Comment:
hive.iceberg.rest.scan-planning-mode {client, server} ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135043752
##########
pom.xml:
##########
@@ -160,6 +159,9 @@
<httpcomponents.core.version>4.4.13</httpcomponents.core.version>
<!-- Upgrade roaringbit version whenever upgrading iceberg version -->
<iceberg.version>1.11.0</iceberg.version>
+ <!-- Jetty 12 for Iceberg REST catalog embedded-server tests (Hive uses jetty.version 9.x elsewhere). -->
+ <iceberg.rest.test.jetty.version>12.1.8</iceberg.rest.test.jetty.version>
Review Comment:
is it misalligned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135255082
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,16 +155,18 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+ String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+ if (tableIdentifier == null) {
Review Comment:
you check tableIdentifier and set executorTableId, that is inconsistent.
maybe drop the executorTableId completely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135828601
##########
pom.xml:
##########
@@ -160,6 +159,9 @@
<httpcomponents.core.version>4.4.13</httpcomponents.core.version>
<!-- Upgrade roaringbit version whenever upgrading iceberg version -->
<iceberg.version>1.11.0</iceberg.version>
+ <!-- Jetty 12 for Iceberg REST catalog embedded-server tests (Hive uses jetty.version 9.x elsewhere). -->
+ <iceberg.rest.test.jetty.version>12.1.8</iceberg.rest.test.jetty.version>
Review Comment:
Two Jetty stacks are required (9 for Hive, 12 for Iceberg REST embedded-server tests). Putting it in the root pom.xml was wrong since it's used only in one test module. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247827362
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ }
+ return table;
+ }
+
+ private static boolean shouldReloadForServerSideScanPlanning(Configuration conf) {
Review Comment:
Design question on where the server-vs-client planning decision lives.
Right now `shouldReloadForServerSideScanPlanning` re-derives it in the Tez AM from catalog name/default catalog, catalog type, the Hive flag and the catalog `scan-planning-mode`, and most of those only exist in the AM because `RestCatalogScanPlanningUtil` copies them from the HS2 session through two hooks (`overlayTableProperties` and `configureJobConf`). If any of that propagation misses, we silently fall back to client-side planning; the IntegrationTest `executorJobConfWithoutPropagationUsesSerializedTable` actually encodes that behavior.
HS2 has all of this at compile time (session config, the loaded table, its current snapshot). Would it be simpler to decide once there and put an explicit marker in the job conf, e.g. `iceberg.mr.server-side-scan-planning=true` plus the snapshot id to pin? Then `getSplits` only checks that flag, and:
- the AM no longer depends on re-deriving catalog type/mode from copied properties,
- propagation is reduced to the connection settings needed for `loadTable` (and can exclude secrets),
- if the flag is set but the reload fails, we can fail or at least WARN instead of silently degrading,
- the pinned snapshot keeps the scan consistent with what the query was compiled against.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
Could we either restrict propagation to an allowlist of non-secret connection keys (type, uri, warehouse, scan-planning-mode, ...) or route `credential` / `token` / OAuth settings through the job secrets path, the same way `configureInputJobCredentials` handles vended credentials?
Please add test validating that credential / token keys stay out of the propagated job properties.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM (see Tanishq [comment](https://github.com/apache/hive/pull/6789#discussion_r4034298144)).
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
Please add test validating that credential / token keys stay out of the propagated job properties.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM.
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313885820
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
The server-vs-client decision is made at HS2 from the client-side `iceberg.catalog.<name>.scan-planning-mode` only (`getScanPlanningMode`), but Iceberg itself (`RESTSessionCatalog.restTableForScanPlanning`) lets the mode returned in `LoadTableResponse` take precedence over the client property. So if the server requires server-side planning and the client property is unset, Iceberg returns a `RESTTable` while Hive skips the marker and plans locally from manifests.
`overlayTableProperties` already loads the table from the catalog; could the marker be set based on what Iceberg resolved there (the loaded table being a server-planning `RESTTable`, gated by the kill switch) instead of re-reading the client property? Note:
- the server's mode is per table, so the marker would need to be keyed per table rather than per catalog;
- `RESTTable` is package-private, so this probably needs a small helper (e.g. in patched-iceberg-core) or a class-name check for now.
- the per-table marker can go into the per-table job properties in `overlayTableProperties`; both read paths copy those into the conf right before `getSplits` (`HiveInputFormat.addSplitsForGroup` for Tez, `FetchOperator` via `copyTableJobPropertiesToConf` for fetch tasks), so `configureJobConf` shouldn't need to write the marker into the shared JobConf at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313924418
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,27 +162,45 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table serializedTable = HiveTableUtil.deserializeTable(conf, tableIdentifier);
+
final ExecutorService workerPool =
ThreadPools.newFixedThreadPool("iceberg-plan-worker-pool",
conf.getInt(SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(), ThreadPools.WORKER_THREAD_POOL_SIZE));
try {
- return planInputSplits(table, conf, workerPool);
+ if (HiveTableUtil.shouldReloadForServerSideScanPlanning(conf, serializedTable)) {
+ String catalogName = HiveTableUtil.resolveCatalogName(conf);
+ try (CloseableRestCatalog catalog = CloseableRestCatalog.load(conf, catalogName)) {
+ Table planningTable = catalog.loadTable(tableIdentifier);
+ HiveTableUtil.checkAndSetIoConfig(conf, planningTable);
+ return planInputSplits(planningTable, conf, workerPool, tableIdentifier, serializedTable);
Review Comment:
**FOLLOW-UP PR**:
On the server path `planInputSplits` gets the live `planningTable`, and `generateInputSplits` builds the grouping key from it: `Partitioning.groupingKeyType(table.schema().select(cols), table.specs().values())`.
Everything else uses the compile-time serialized table.
If a grouping column is renamed/dropped, or a spec without that field is added between compile and split time, this either fails or changes the grouping key.
Could `generateInputSplits` use `serverSidePinTable` (when non-null) for schema/specs? -
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201810435
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201794299
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
Renamed catalog helpers to `setCatalogMode` / `getCatalogMode` / `isCatalogServerSidePlanningMode` so they’re distinct from the Hive `setHiveMode` / `getHiveMode` / isHiveServerModeEnabled` APIs and the boolean matches the scan-planning property semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247563182
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setCatalogMode(Configuration, String, String)},
+ * {@link #isCatalogServerMode(Configuration, String)}, and
+ * {@link #setHiveMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ private static void setCatalogMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ mode.modeName());
+ }
+
+ public static void setCatalogMode(Configuration conf, String catalogName, String mode) {
+ setCatalogMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ static RESTCatalogProperties.ScanPlanningMode getCatalogMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isCatalogServerMode(Configuration conf, String catalogName) {
Review Comment:
Server mode needs both hive.iceberg.rest.scan-planning-mode=server and the per-catalog scan-planning-mode=server
The REST server can advertise the planning mode in LoadTableResponse.config, which is exactly what the IntegrationTests simulate, so the Hive-side check can disagree with what the loaded table will actually do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM.
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300196587
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ }
+ return table;
+ }
+
+ private static boolean shouldReloadForServerSideScanPlanning(Configuration conf) {
Review Comment:
Checked against Iceberg 1.12: `RESTTable` only overrides `newScan()`, so snapshot id, as-of and refs go through `RESTTableScan`. Incremental append scans and metadata tables are planned locally even on a `RESTTable`, so `shouldReloadForServerSideScanPlanning` now excludes them. On the server path, ref names resolve against the serialized table so they can't drift either. Added ITs for snapshot id, tag, and incremental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4291289087
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4289579218
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
Thanks — all three blockers are addressed:
- Snapshot drift — In getSplits, when server mode applies and there’s no explicit snapshot/ref/as-of, we pin the REST scan to the serialized compile-time table (pinServerSideScanToSerializedTable). Tests: `TestServerSideScanSnapshotPinning`, `TestHiveIcebergServerSideScanPlanningServerIT#serverSidePlanningPinsSerializedSnapshotAfterCatalogAdvances`.
- Secrets — Non-secret catalog props → job properties; credential/token/oauth (and hidden keys) → `configureInputJobCredentials`. Test: `propagateInputJobCredentialsRoutesCredentialAndOAuthToJobSecrets`.
- Catalog leak — using `try (CloseableRestCatalog …)` in `getSplits`. catalog is closed after `planTasks()` materializes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313924418
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -157,27 +162,45 @@ private static <T extends Scan<T, FileScanTask, CombinedScanTask>> T applyConfig
@Override
public List<InputSplit> getSplits(JobContext context) {
Configuration conf = context.getConfiguration();
- Table table = Optional
- .ofNullable(HiveTableUtil.deserializeTable(conf, conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
- .orElseGet(() -> {
- Table tbl = Catalogs.loadTable(conf);
- conf.set(InputFormatConfig.TABLE_IDENTIFIER, tbl.name());
- // planning-local conf only (never shipped): for credential-vending catalogs the loaded
- // table's FileIO carries secrets, which must not reach a serialized job configuration
- conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tbl.name(), SerializationUtil.serializeToBase64(tbl));
- return tbl;
- });
+ String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+ Table serializedTable = HiveTableUtil.deserializeTable(conf, tableIdentifier);
+
final ExecutorService workerPool =
ThreadPools.newFixedThreadPool("iceberg-plan-worker-pool",
conf.getInt(SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(), ThreadPools.WORKER_THREAD_POOL_SIZE));
try {
- return planInputSplits(table, conf, workerPool);
+ if (HiveTableUtil.shouldReloadForServerSideScanPlanning(conf, serializedTable)) {
+ String catalogName = HiveTableUtil.resolveCatalogName(conf);
+ try (CloseableRestCatalog catalog = CloseableRestCatalog.load(conf, catalogName)) {
+ Table planningTable = catalog.loadTable(tableIdentifier);
+ HiveTableUtil.checkAndSetIoConfig(conf, planningTable);
+ return planInputSplits(planningTable, conf, workerPool, tableIdentifier, serializedTable);
Review Comment:
on the server path `planInputSplits` gets the live `planningTable`, and `generateInputSplits` builds the grouping key from it: `Partitioning.groupingKeyType(table.schema().select(cols), table.specs().values())`.
Everything else uses the compile-time serialized table. If a grouping column is renamed/dropped, or a spec without that field is added between compile and split time, this either fails or changes the grouping key.
Could `generateInputSplits` use `serverSidePinTable` (when non-null) for schema/specs? - follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313652650
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
For `t.branch_x`, `OUTPUT_TABLE_SNAPSHOT_REF` is set, `hasExplicitScanSnapshotBinding` returns true, and `scan.useRef(...)` resolves the branch on the live `RESTTable` in the AM.
Could we resolve that ref against the serialized table too (as done for `SNAPSHOT_ID` names) and add an IT that advances the branch after compile?
````
String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
if (StringUtils.isNotEmpty(refName)) {
String branch = HiveUtils.getTableSnapshotRef(refName);
if (serverSidePinTable != null) {
SnapshotRef ref = serverSidePinTable.refs().get(branch);
Preconditions.checkArgument(ref != null, "Cannot find ref %s", branch);
scan = scan.useSnapshot(ref.snapshotId());
} else {
scan = scan.useRef(branch);
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313588904
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
+
+ private final Catalog catalog;
+
+ public static CloseableRestCatalog load(Configuration conf, String catalogName) {
+ return Catalogs.loadCatalog(conf, catalogName)
+ .map(CloseableRestCatalog::new)
+ .orElseThrow(
+ () ->
+ new IllegalStateException(
+ "Iceberg catalog is required for REST server-side scan planning"));
+ }
+
+ private CloseableRestCatalog(Catalog catalog) {
+ this.catalog = catalog;
+ }
+
+ public Table loadTable(String tableIdentifier) {
+ return catalog.loadTable(TableIdentifier.parse(tableIdentifier));
+ }
+
+ @Override
+ public void close() {
+ if (catalog instanceof AutoCloseable closeable) {
+ try {
+ closeable.close();
+ } catch (Exception e) {
+ throw new UncheckedIOException(
Review Comment:
should we throw exception or simply LOG.warn("Failed to close Iceberg catalog {}", catalog.name(), e);?
close failure kills a query that already planned successfully
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
Instead of fixing `close()` here, could we drop `CloseableRestCatalog` and use Iceberg's `org.apache.iceberg.io.CloseableGroup` in `getSplits`?
It closes any `AutoCloseable` (`RESTCatalog` is `Closeable`) and with `setSuppressCloseFailure(true)` logs close failures instead of throwing:
````
try (CloseableGroup closer = new CloseableGroup()) {
closer.setSuppressCloseFailure(true);
if (catalog instanceof AutoCloseable closeable) {
closer.addCloseable(closeable);
}
...
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313671696
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ /**
+ * Returns true when Hive is allowed to use server-side scan planning (the Hive-side kill switch).
+ */
+ static boolean isServerSideScanPlanningEnabled(Configuration conf) {
+ return conf != null &&
+ HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog's scan planning mode is {@code server} and the Hive kill switch is on.
+ */
+ static boolean shouldUseServerSideScanPlanning(Configuration conf, String catalogName) {
+ if (StringUtils.isEmpty(catalogName) || !isServerSideScanPlanningEnabled(conf)) {
+ return false;
+ }
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return shouldUseServerSideScanPlanning(conf, resolvedCatalogName);
+ }
+
+ /**
+ * Copies non-secret {@code iceberg.catalog.<catalog>.*} entries from the HS2 session into Tez job
+ * properties, together with the {@link InputFormatConfig#serverSideScanPlanningKey(String)} marker.
+ * Sensitive catalog settings are copied via
+ * {@link #propagateInputJobCredentials(TableDesc, Map, Configuration)} (HIVE-20651 job secrets).
+ * Session values replace existing entries so a session {@code SET} wins.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobProperties::put, null);
+ }
+
+ /**
+ * Copies sensitive {@code iceberg.catalog.<catalog>.*} entries into job secrets for
+ * {@link org.apache.hadoop.hive.ql.metadata.HiveStorageHandler#configureInputJobCredentials}.
+ */
+ static void propagateInputJobCredentials(
+ TableDesc tableDesc, Map<String, String> jobSecrets, Configuration conf) {
+ if (tableDesc == null || tableDesc.getProperties() == null || jobSecrets == null || conf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(
+ conf,
+ tableDesc.getProperty(InputFormatConfig.CATALOG_NAME),
+ null,
+ jobSecrets::put);
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet. Secrets are not written to {@code jobConf}.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogPropertiesToJob(sessionConf, catalogName, jobConf::set, null);
+ }
+
+ private static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf,
+ String catalogName,
+ BiConsumer<String, String> jobPropertySink,
+ BiConsumer<String, String> jobSecretSink) {
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ String resolvedCatalogName = HiveTableUtil.resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (jobPropertySink != null) {
+ jobPropertySink.accept(
+ InputFormatConfig.serverSideScanPlanningKey(resolvedCatalogName), Boolean.TRUE.toString());
+
+ String sessionDefaultCatalog =
+ MetastoreConf.getVar(sessionConf, MetastoreConf.ConfVars.CATALOG_DEFAULT);
+ if (StringUtils.isNotBlank(sessionDefaultCatalog)) {
+ jobPropertySink.accept(MetastoreConf.ConfVars.CATALOG_DEFAULT.getVarname(), sessionDefaultCatalog);
+ }
+ }
+
+ String catalogPrefix =
+ IcebergCatalogProperties.CATALOG_CONFIG_PREFIX + resolvedCatalogName + ".";
+ sessionConf.forEach(
+ entry -> {
+ if (!entry.getKey().startsWith(catalogPrefix)) {
+ return;
+ }
+ String catalogPropertySuffix = entry.getKey().substring(catalogPrefix.length());
+ if (isSecretCatalogConfigKey(entry.getKey(), catalogPropertySuffix, sessionConf)) {
+ if (jobSecretSink != null) {
+ jobSecretSink.accept(entry.getKey(), entry.getValue());
+ }
+ } else if (jobPropertySink != null) {
+ jobPropertySink.accept(entry.getKey(), entry.getValue());
+ }
+ });
+ }
+
+ static boolean isSecretCatalogConfigKey(
+ String fullConfigKey, String catalogPropertySuffix, Configuration sessionConf) {
+ if (IcebergVendedCredentialUtil.isSecretKey(fullConfigKey, sessionConf)) {
+ return true;
+ }
+ return isSecretCatalogPropertySuffix(catalogPropertySuffix);
+ }
+
+ private static boolean isSecretCatalogPropertySuffix(String catalogPropertySuffix) {
+ if (StringUtils.isBlank(catalogPropertySuffix)) {
+ return false;
+ }
+ String lower = catalogPropertySuffix.toLowerCase(Locale.ROOT);
+ return lower.contains("secret") || lower.contains("token") || lower.contains("password") ||
Review Comment:
On `isSecretCatalogPropertySuffix`: a substring denylist both under-matches (`header.Authorization` / `header.X-Api-Key`, custom `AuthManager` properties) and over-matches (`oauth2-server-uri`, `token-refresh-enabled`).
Since only the AM's `loadCatalog` needs these properties (executors read through the serialized table, and the commit runs in HS2 on the session conf) and job secrets already reach split generation via `addSplitsForGroup`, could we route all `iceberg.catalog.<name>.*` keys through job secrets and keep only the marker and `metastore.catalog.default` in plain job properties? That removes the classification entirely. Note that `configureJobConf` currently writes the catalog keys directly into the JobConf, so it would need to set only the marker as well.
If you'd rather keep some keys visible, please flip to an allowlist of known non-sensitive keys (type, uri, warehouse, catalog-impl, io-impl, prefix, scan-planning-mode) so unknown keys default to secrets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4314127905
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
wonder if we should propose to upstream it in iceberg? but i think it should be moved to the catalog module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6333954289
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135093112
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestRestCatalogScanPlanningServerIT.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.rest.requests.PlanTableScanRequest;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/**
+ * Embedded REST server tests for {@link RestCatalogScanPlanning}: when Hive-style configuration sets
+ * {@code scan-planning-mode=server}, {@link RESTTable} / {@link RESTTableScan} delegate split
+ * planning to the catalog server ({@code POST /plan}).
+ */
+class TestRestCatalogScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
don't we use HiveRESTCatalogServerExtension restCatalogExtension = getHiveRESTCatalogServerExtension() do decorate with additional fixtures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135619721
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -2154,6 +2154,12 @@ public static enum ConfVars {
"If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
"If this is set to true, then all the data files being read should be withing the table location"),
+ HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED(
Review Comment:
Renamed to `hive.iceberg.rest.scan-planning-mode` with client/server to mirror Iceberg REST catalog property naming; default client preserves prior opt-in behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135749861
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -343,6 +344,10 @@ public void commitJob(JobContext originalContext) {
public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
setCommonJobConf(jobConf);
configureOutputTableJobConf(tableDesc, jobConf);
+ if (tableDesc != null && tableDesc.getProperties() != null) {
+ String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
Review Comment:
If we move this to If we move this ti tableDesc we will add Iceberg-specific code to hive-exec module.
How about `tableDesc.getProperty(InputFormatConfig.CATALOG_NAME)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4136350032
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
Review Comment:
Moved this class to iceberg-handler (because it's not rest client specific) and added Util to the class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5903937166
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [103 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [1.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164811129
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
Review Comment:
getScanPlanningMode vs getHiveRestScanPlanningMode - this add confusion.
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
Review Comment:
same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4164804973
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
naming not consistent with rest of public methods, like `getScanPlanningMode`. should we align?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4382258302
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
Done. A missing ref in the compile-time table now fails via `Preconditions.checkState` instead of falling back to a live `useRef`. For the older-schema branch case we still use `useRef` (there's no public API to pin a snapshot while keeping the table schema), but we fail fast with "Branch %s moved since query compilation" if the live head differs from the compile-time ref. Added tests for a missing ref, a moved branch, and tag pinning via `useSnapshot`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add REST catalog server-side scan planning support [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5695110013
There were already discussions happening under [HIVE-29599](https://issues.apache.org/jira/browse/HIVE-29599) for server side planning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4389222582
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
did you push?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6528222163
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135214729
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!RestCatalogScanPlanning.shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
Review Comment:
please check visibility modifiers in public methods if the actually need thos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135146875
##########
iceberg/iceberg-rest-catalog-client/src/main/java/org/apache/iceberg/hive/rest/catalog/RestCatalogScanPlanning.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
Review Comment:
RestCatalogScanPlanningUtil/s ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135229245
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanning.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.nio.file.Path;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Unit tests for Hive server-side REST catalog scan planning via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}. Embedded REST server coverage is in
+ * {@code TestHiveIcebergServerSideScanPlanningServerIT} in {@code itests/hive-iceberg-rest-server}.
+ */
+class TestHiveIcebergServerSideScanPlanning {
Review Comment:
does it test ServerSideScanPlanning? usesDeserializedTable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4136232365
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanningServerIT.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
We use `HiveRESTCatalogServerExtension` for tests that hit the HMS-backed REST catalog servlet. These scan-planning Server ITs intentionally use Iceberg’s `TestBaseWithRESTServer`, an in-process Iceberg REST harness, not the HMS servlet, so we can spy `RESTCatalogAdapter`, drive `HiveTableUtil.resolveTableForScanPlanning` / job-conf propagation, and keep the unshaded Iceberg + Jetty 12 classpath this module needs. Different server, different layer under test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4136275379
##########
itests/hive-iceberg-rest-server/src/test/java/org/apache/iceberg/rest/TestHiveIcebergServerSideScanPlanningServerIT.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.Consumer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Scan;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.hive.rest.catalog.RestCatalogScanPlanning;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.mr.hive.HiveTableUtil;
+import org.apache.iceberg.rest.responses.ErrorResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+
+import static org.apache.iceberg.TestBase.FILE_A;
+import static org.apache.iceberg.TestBase.SCHEMA;
+import static org.apache.iceberg.TestBase.SPEC;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {
Review Comment:
Test class. It’s a concrete JUnit tests. It uses the framework base `TestBaseWithRESTServer` (from iceberg-core tests); it isn’t a reusable fixture or extension itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247674427
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
`hive.iceberg.rest.server-side-scan-planning.enabled` = true | false.
<p dir="ltr">The helper methods then each answer one question:</p>
<div dir="ltr">
Method | Answers
-- | --
getScanPlanningMode(conf, catalogName) → ScanPlanningMode | What does the catalog want?
isServerSideScanPlanningEnabled(conf) | Has Hive disabled the feature?
shouldUseServerSideScanPlanning(conf, catalogName) | Mode is SERVER and the switch is on
</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247714315
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
- `hive.iceberg.rest.server-side-scan-planning.enabled` (boolean)
- the mode itself comes only from the catalog / what Iceberg resolves on the loaded table
Helpers then become:
- `getScanPlanningMode(conf, catalogName)` → `ScanPlanningMode`
- `isServerSideScanPlanningEnabled(conf)` → the kill switch
- `shouldUseServerSideScanPlanning(conf, catalogName)` → mode is `SERVER` and the switch is on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247674427
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
Review Comment:
my bad, could we please reintroduce Hive-side feature flag, so naming stays clean. The catalog (or what Iceberg resolves on the loaded table) decides the mode. The Hive setting only says whether Hive is allowed to follow it.
- `hive.iceberg.rest.server-side-scan-planning.enabled` (boolean)
- the mode itself comes only from the catalog / what Iceberg resolves on the loaded table
Helpers then become:
- `getScanPlanningMode(conf, catalogName)` → `ScanPlanningMode`
- `isServerSideScanPlanningEnabled(conf)` → the kill switch
- `shouldUseServerSideScanPlanning(conf, catalogName)` → mode is `SERVER` and the switch is on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4247525890
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +262,57 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
Review Comment:
1. The query can read a different snapshot than the one it compiled against.
HS2 compiles against snapshot N and serializes that table into the job conf. In server mode, split generation runs later in the Tez AM and calls Catalogs.loadTable, which returns whatever is current then (N+k)
The TABLE_METADATA_LOCATION guard covers read-after-write inside a transaction, but not this drift.
The fix is cheap: when no explicit snapshot, ref, or as-of time is set, pin the REST scan to `serializedTable.currentSnapshot().snapshotId()` and project the serialized schema. The live table is then used only to send POST /plan.
please add test for the drift case: serialize the table, commit a new snapshot through the catalog, then call `resolveTableForScanPlanning` / `getSplits` and assert the planned tasks belong to the serialized snapshot, not the new one
2. Catalog properties, including secrets, are copied into shipped job properties.
`propagateCatalogPropertiesToJob` copies every iceberg.catalog.<name>.* entry. That includes credential, token, and OAuth client secrets if they are configured. In overlayTableProperties they go into the TableDesc job-properties map, which is serialized into the plan and job.xml.
Could we either restrict propagation to an allowlist of non-secret connection keys (type, uri, warehouse, scan-planning-mode, ...) or route `credential` / `token` / OAuth settings through the job secrets path, the same way `configureInputJobCredentials` handles vended credentials?
Please add test validating that credential / token keys stay out of the propagated job properties.
3. Each reload leaks a catalog and its HTTP client in a long-lived AM (see Tanishq [comment](https://github.com/apache/hive/pull/6789#discussion_r4034298144)).
Here it runs in the Tez AM, which is reused across queries with session pooling and LLAP. With OAuth2, the REST session can also own a token-refresh executor. One unclosed RESTCatalog per scan per query adds up to leaked threads and connections.
getSplits already materializes planTasks() into a list, so the catalog can be closed right after planning (try-with-resources around load, plan, and materialize). The other option is a bounded per-catalog cache in the AM.
Snapshot pinning and secret filtering are blockers, and catalog lifecycle should at least be fixed for the AM path.
suggested shape:
````
Table snapshotView = deserializeTable(conf, id); // compile-time truth (schema, specs, snapshot)
if (serverPlanningRequested(conf) && snapshotView != null && !intraTxn(conf)) {
try (RestPlanningSession s = RestPlanningSession.open(conf, catalogName)) { // closes catalog/HTTP client
TableScan scan = s.load(id).newScan()
.useSnapshot(pinnedSnapshotId(conf, snapshotView))
.project(snapshotView.schema());
// apply filters/options as today, then materialize planTasks() before close
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-5804906554
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [29 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4135749861
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -343,6 +344,10 @@ public void commitJob(JobContext originalContext) {
public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
setCommonJobConf(jobConf);
configureOutputTableJobConf(tableDesc, jobConf);
+ if (tableDesc != null && tableDesc.getProperties() != null) {
+ String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
Review Comment:
If we move this to `tableDesc` we will add Iceberg-specific code to hive-exec module.
How about `tableDesc.getProperty(InputFormatConfig.CATALOG_NAME)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4190125059
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.hive.rest.catalog;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link org.apache.iceberg.mr.hive.HiveTableUtil#resolveTableForScanPlanning}, which reloads the
+ * live REST catalog table (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} is true.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanning} in {@code iceberg-rest-catalog-client};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {
+
+ private RestCatalogScanPlanning() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED);
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
Review Comment:
Good catch. It has almost the same purpose as `HiveTableUtil.catalogNameForScan` method. Consolidated into an overloaded `resolveCatalogName` method in `HiveTableUtil`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub) via gitbox" <gi...@hive.apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6533932657
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4201797721
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
Review Comment:
This has been addressed - now it is `getCatalogMode` (scan-planning mode in catalog properties) vs `getHiveMode` (Hive feature flag)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4202035742
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@link HiveTableUtil#resolveTableForScanPlanning}, which reloads the live REST catalog table
+ * (instead of a serialized metadata snapshot) when server mode is enabled and
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SCAN_PLANNING_MODE} is {@code server}.
+ * Operators can use this helper or set catalog {@code scan-planning-mode} directly in {@code hive-site.xml}.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static String catalogPropertyKey(String catalogName) {
+ return IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE);
+ }
+
+ public static void setScanPlanningMode(
+ Configuration conf, String catalogName, RESTCatalogProperties.ScanPlanningMode mode) {
+ conf.set(catalogPropertyKey(catalogName), mode.modeName());
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
+ Configuration conf, String catalogName) {
+ String mode = conf.get(
+ catalogPropertyKey(catalogName), RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName());
+ return RESTCatalogProperties.ScanPlanningMode.fromString(mode);
+ }
+
+ public static boolean isServerMode(Configuration conf, String catalogName) {
+ return getScanPlanningMode(conf, catalogName) == RESTCatalogProperties.ScanPlanningMode.SERVER;
+ }
+
+ /**
+ * Returns true when Hive server-side REST scan planning is enabled in configuration.
+ */
+ public static boolean isHiveServerSideScanPlanningEnabled(Configuration conf) {
+ if (conf == null) {
+ return false;
+ }
+ return RESTCatalogProperties.ScanPlanningMode.SERVER ==
+ RESTCatalogProperties.ScanPlanningMode.fromString(getHiveRestScanPlanningMode(conf));
+ }
+
+ public static String getHiveRestScanPlanningMode(Configuration conf) {
+ if (conf == null) {
+ return RESTCatalogProperties.SCAN_PLANNING_MODE_DEFAULT.modeName();
+ }
+ return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+ }
+
+ public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {
+ HiveConf.setVar(
+ conf,
+ HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ /**
+ * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+ */
+ public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {
+ if (conf == null || StringUtils.isEmpty(catalogName)) {
+ return false;
+ }
+ return isHiveServerSideScanPlanningEnabled(conf) && isServerMode(conf, catalogName);
+ }
+
+ /**
+ * Returns true when catalog properties should be copied into the Tez/MR job configuration so
+ * executors can reload a live REST catalog table for server-side scan planning.
+ */
+ public static boolean shouldPropagateCatalogPropertiesToJob(String catalogName, Configuration conf) {
+ String resolvedCatalogName = resolveCatalogName(conf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName) || conf == null) {
+ return false;
+ }
+ if (!CatalogUtil.ICEBERG_CATALOG_TYPE_REST.equals(
+ IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+ return false;
+ }
+ return requestsServerSidePlanning(resolvedCatalogName, conf);
+ }
+
+ /**
+ * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+ */
+ public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {
+ if (StringUtils.isNotBlank(catalogNameFromTable)) {
+ return catalogNameFromTable;
+ }
+ if (conf == null) {
+ return null;
+ }
+ return IcebergCatalogProperties.getCatalogName(conf);
+ }
+
+ /**
+ * Copies {@code iceberg.catalog.<catalog>.*} entries from the HS2 session configuration into Tez/MR
+ * job properties so executors can reload a live REST catalog table for server-side scan planning.
+ *
+ * <p>Session-level {@code SET} commands and {@code hive-site.xml} catalog settings are not
+ * automatically present in the job configuration; without this step split generation falls back to
+ * the serialized metadata snapshot ({@code DataTableScan}).
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Map<String, String> jobProperties) {
+ if (sessionConf == null || jobProperties == null) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {
+ return;
+ }
+
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> jobProperties.putIfAbsent(key, value));
+ }
+
+ /**
+ * Copies REST catalog configuration from the HS2 session into a runtime job {@link Configuration}.
+ * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+ * even when job properties were not copied yet.
+ */
+ public static void propagateCatalogPropertiesToJob(
+ Configuration sessionConf, String catalogName, Configuration jobConf) {
+ if (sessionConf == null || jobConf == null) {
+ return;
+ }
+ propagateCatalogProperties(sessionConf, catalogName, (key, value) -> {
+ if (jobConf.get(key) == null) {
+ jobConf.set(key, value);
+ }
+ });
+ }
+
+ private static void propagateCatalogProperties(
+ Configuration sessionConf, String catalogName, PropertyConsumer consumer) {
+ String resolvedCatalogName = resolveCatalogName(sessionConf, catalogName);
+ if (StringUtils.isEmpty(resolvedCatalogName)) {
+ return;
+ }
+
+ if (!shouldPropagateCatalogPropertiesToJob(resolvedCatalogName, sessionConf)) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313652650
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
For `t.branch_x`, `OUTPUT_TABLE_SNAPSHOT_REF` is set, `hasExplicitScanSnapshotBinding` returns true, and `scan.useRef(...)` resolves the branch on the live `RESTTable` in the AM. Could we resolve that ref against the serialized table too (as done for `SNAPSHOT_ID` names) and add an IT that advances the branch after compile?
````
String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
if (StringUtils.isNotEmpty(refName)) {
String branch = HiveUtils.getTableSnapshotRef(refName);
if (serverSidePinTable != null) {
SnapshotRef ref = serverSidePinTable.refs().get(branch);
Preconditions.checkArgument(ref != null, "Cannot find ref %s", branch);
scan = scan.useSnapshot(ref.snapshotId());
} else {
scan = scan.useRef(branch);
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4314127905
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/CloseableRestCatalog.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * {@link Catalog} loaded from job configuration and closed after use when the implementation
+ * supports {@link AutoCloseable}.
+ *
+ * <p>Used for short-lived catalog access (e.g. REST server-side scan planning in split generation)
+ * so HTTP clients and OAuth refresh executors are not leaked on long-lived Tez AMs.
+ */
+public final class CloseableRestCatalog implements AutoCloseable {
Review Comment:
Instead of fixing `close()` here, could we drop `CloseableRestCatalog` and use Iceberg's `org.apache.iceberg.io.CloseableGroup` in `getSplits`? It closes any `AutoCloseable` (`RESTCatalog` is `Closeable`) and with `setSuppressCloseFailure(true)` logs close failures instead of throwing:
````
try (CloseableGroup closer = new CloseableGroup()) {
closer.setSuppressCloseFailure(true);
if (catalog instanceof AutoCloseable closeable) {
closer.addCloseable(closeable);
}
...
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4313885820
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/RestCatalogScanPlanningUtil.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.rest.RESTCatalogProperties;
+
+/**
+ * Utilities for Iceberg REST catalog server-side scan planning configuration.
+ *
+ * <p>When a REST catalog server advertises the scan-planning endpoints and
+ * {@link RESTCatalogProperties#SCAN_PLANNING_MODE} is set to
+ * {@link RESTCatalogProperties.ScanPlanningMode#SERVER}, Iceberg's {@code RESTSessionCatalog} returns a
+ * {@code RESTTable} that delegates {@code planTasks()} to the server. Hive's
+ * {@code IcebergInputFormat} calls {@code scan.planTasks()} via
+ * {@code IcebergInputFormat#getSplits}, which reloads a live REST catalog table for {@code POST /plan}
+ * (pinned to the serialized compile-time snapshot). The catalog decides the scan planning mode;
+ * {@link HiveConf.ConfVars#HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED} only controls whether
+ * Hive is allowed to follow it.
+ *
+ * <p>Tests: {@code TestRestCatalogScanPlanningUtil} in {@code iceberg-handler};
+ * {@code TestHiveIcebergServerSideScanPlanning} in {@code iceberg-handler}; embedded REST server
+ * tests {@code TestRestCatalogScanPlanningServerIT} and {@code TestHiveIcebergServerSideScanPlanningServerIT}
+ * in {@code itests/hive-iceberg-rest-server}.
+ *
+ * <p>{@link #setScanPlanningMode(Configuration, String, String)} and
+ * {@link #getScanPlanningMode(Configuration, String)} are public so tests in other Maven modules
+ * (for example {@code itests/hive-iceberg-rest-server}) can configure scan planning; those modules
+ * compile against this artifact as a JAR and cannot call package-private members. They are not
+ * intended as a general operator or application API.
+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanningUtil {
+
+ private RestCatalogScanPlanningUtil() {
+ }
+
+ public static void setScanPlanningMode(Configuration conf, String catalogName, String mode) {
+ conf.set(
+ IcebergCatalogProperties.catalogPropertyConfigKey(
+ catalogName, RESTCatalogProperties.SCAN_PLANNING_MODE),
+ RESTCatalogProperties.ScanPlanningMode.fromString(mode).modeName());
+ }
+
+ public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(
Review Comment:
The server-vs-client decision is made at HS2 from the client-side `iceberg.catalog.<name>.scan-planning-mode` only (`getScanPlanningMode`), but Iceberg itself (`RESTSessionCatalog.restTableForScanPlanning`) lets the mode returned in `LoadTableResponse` take precedence over the client property. So if the server requires server-side planning and the client property is unset, Iceberg returns a `RESTTable` while Hive skips the marker and plans locally from manifests.
`overlayTableProperties` already loads the table from the catalog; could the marker be set based on what Iceberg resolved there (the loaded table being a server-planning `RESTTable`, gated by the kill switch) instead of re-reading the client property? Note:
- the server's mode is per table, so the marker would need to be keyed per table rather than per catalog;
- `RESTTable` is package-private, so this probably needs a small helper (e.g. in patched-iceberg-core) or a class-name check for now.
- the per-table marker can simply go into the per-table job properties in `overlayTableProperties` (`addSplitsForGroup` copies those into the conf before that table's `getSplits`); `configureJobConf` writes to the shared JobConf, so there it would need to be keyed by table identifier or dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4314074254
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -278,17 +278,8 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
- if (!IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
- return;
- }
- try {
- Table table =
- IcebergVendedCredentialUtil.getTableWithVendedCredentials(tableDesc.getProperties(), conf);
- String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
- IcebergVendedCredentialUtil.propagateToJob(table, catalogName, null, secrets, conf);
- } catch (NoSuchTableException ex) {
- // Table may not exist yet for CTAS; credentials will not be available.
- }
+ RestCatalogScanPlanningUtil.propagateInputJobCredentials(tableDesc, secrets, conf);
Review Comment:
two `propagateInputJobCredentials` calls with the same name but different jobs: one copies catalog auth config (`iceberg.catalog.<name>.*` secrets) so the AM can reach the REST catalog, the other copies vended storage credentials for the readers.
Could we name them after what they carry, e.g. `RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob` (pairs with `propagateCatalogPropertiesToJob`) and `IcebergVendedCredentialUtil.propagateVendedCredentialsToJob`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4375378312
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -72,31 +72,36 @@
*/
public class IcebergInputFormat<T> extends InputFormat<Void, T> {
- private static TableScan createTableScan(Table table, Configuration conf) {
+ private static TableScan createTableScan(Table table, Configuration conf, Table serverSidePinTable) {
TableScan scan = table.newScan();
- long snapshotId = -1;
- try {
- snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
- } catch (NumberFormatException e) {
- String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
- SnapshotRef ref = table.refs().get(version);
- if (ref == null) {
- throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ if (HiveTableUtil.hasExplicitScanSnapshotBinding(conf)) {
+ long snapshotId = -1;
+ try {
+ snapshotId = conf.getLong(InputFormatConfig.SNAPSHOT_ID, -1);
+ } catch (NumberFormatException e) {
+ String version = conf.get(InputFormatConfig.SNAPSHOT_ID);
+ Table refsTable = serverSidePinTable != null ? serverSidePinTable : table;
+ SnapshotRef ref = refsTable.refs().get(version);
+ if (ref == null) {
+ throw new RuntimeException("Cannot find matching snapshot ID or reference name for version " + version);
+ }
+ snapshotId = ref.snapshotId();
+ }
+ String refName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ if (StringUtils.isNotEmpty(refName)) {
Review Comment:
the `useRef` exception for branches with an older schema makes sense since there's no public API to pin a snapshot while keeping the table schema. But in that case a commit to the branch between compile and split time still changes what we read. Could we fail fast there instead of silently reading the new head, e.g.
````
TableScan pinned = scan.useRef(refName);
Preconditions.checkState(pinned.snapshot().snapshotId() == ref.snapshotId(),
"Branch %s moved since query compilation", refName);
````
Also, when the ref is missing from the serialized table we still fall back to the live `useRef`. Could that throw, as in the original suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4300410648
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -259,6 +261,71 @@ public static Table deserializeTable(Configuration config, String name) {
return table;
}
+ /**
+ * Resolves the Iceberg {@link Table} for split generation ({@code IcebergInputFormat#getSplits}).
+ *
+ * <p>Serialized tables ({@link SerializableTable}) only carry a metadata snapshot and produce
+ * {@link org.apache.iceberg.DataTableScan} (client-side manifest planning). When REST catalog
+ * server-side scan planning is enabled, reload the live table from the catalog so
+ * {@code table.newScan()} returns {@link org.apache.iceberg.rest.RESTTableScan} and issues
+ * {@code POST /plan} on the REST server.
+ *
+ * <p>Intra-transaction read-after-write ({@link InputFormatConfig#TABLE_METADATA_LOCATION}) still
+ * uses the deserialized snapshot so uncommitted metadata is visible.
+ */
+ public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+ if (shouldReloadForServerSideScanPlanning(conf)) {
+ Table table = Catalogs.loadTable(conf);
+ checkAndSetIoConfig(conf, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, resolveCatalogName(conf), conf);
+ return table;
+ }
+
+ Table table = deserializeTable(conf, tableIdentifier);
+ if (table == null) {
+ table = Catalogs.loadTable(conf);
Review Comment:
Simplified instead of unifying: removed `resolveTableForScanPlanning`. The only catalog load the PR adds is the server-side reload in `getSplits` (inside `try (CloseableRestCatalog …))`, and it no longer calls `applyFromJobConf`. That table is only used for POST /plan, which is pure REST, so the AM doesn't read storage with it. The no-serialized-table fallback is back to master's `Catalogs.loadTable(conf)`. So both load paths now treat credentials the same way.
On your two questions:
- applyFromJobConf is a no-op when there are no serialized creds and no delegation header. It's no longer called on either load path, only from `deserializeTable` as before.
- Executors read through the deserialized compile-time table, so the creds HS2 vended at compile time are the ones used. Creds on the AM's fresh `LoadTableResponse` don't reach the readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6789:
URL: https://github.com/apache/hive/pull/6789#issuecomment-6450543459
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6789&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6789&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6789&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6789)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6789:
URL: https://github.com/apache/hive/pull/6789#discussion_r4422608954
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -279,8 +279,23 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
@Override
public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {
- RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(tableDesc, secrets, conf);
- IcebergVendedCredentialUtil.propagateVendedCredentialsToJob(tableDesc, secrets, conf);
+ Properties props = tableDesc.getProperties();
+ if (props == null) {
+ return;
+ }
+ if (!IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf) &&
+ !RestCatalogScanPlanningUtil.isServerSideScanPlanningEnabled(conf)) {
+ return;
+ }
+ Table table;
+ try {
+ table = IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, conf);
+ } catch (NoSuchTableException e) {
+ // Table may not exist yet for CTAS; there is nothing to plan or vend credentials for.
+ return;
+ }
+ RestCatalogScanPlanningUtil.propagateCatalogSecretsToJob(table, tableDesc, secrets, conf);
Review Comment:
thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org