You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "difin (via GitHub)" <gi...@apache.org> on 2026/05/11 18:08:23 UTC

[PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog C… [hive]

difin opened a new pull request, #6474:
URL: https://github.com/apache/hive/pull/6474

   …lient
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   - Add Iceberg REST access delegation support in Hive: map `rest.access-delegation` to the `X-Iceberg-Access-Delegation` header (with an escape hatch for an explicit `header.X-Iceberg-Access-Delegation` property) so catalogs can return vended storage credentials per the Iceberg REST spec.
   
   - Extend Gravitino LLAP qtest (`TestIcebergRESTCatalogGravitinoLlapLocalCliDriver`) to run against MinIO + S3 warehouse with Gravitino s3-secret-key vending and OAuth2; configure host-side S3A and Iceberg S3FileIO for the published MinIO port so Tez/LLAP on the host work reliably.
   
   - Add a small RESTCatalog smoke check in test setup (vended S3 / S3FileIO) before the q-files run.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   - Hive’s Iceberg REST client must send the delegation header so Gravitino (and similar catalogs) can attach vended S3 configuration to responses; without it, vended-credential flows cannot be used from Hive as intended.
   
   - The Gravitino integration test needs a realistic S3 + vending layout and host-safe MinIO/Tez configuration so CI does not depend on brittle host hacks (e.g. resolving minio only inside Docker).
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   Yes. Users configuring an Iceberg REST catalog in Hive can set `iceberg.catalog.<name>.rest.access-delegation` (e.g. vended-credentials); Hive will pass that through as `X-Iceberg-Access-Delegation` on REST requests unless they set the header property explicitly. No change for catalogs that do not use this property.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   - TestHiveRESTCatalogClient (unit) for delegation / header mapping.
   
   - TestIcebergRESTCatalogGravitinoLlapLocalCliDriver (LLAP qtest: iceberg_rest_catalog_gravitino.q) with Gravitino + MinIO + OAuth2 + vending.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `iceberg-rest-catalog-client` and move it there including things like HiveRESTCatalogClient, VendedCredentials, etc ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559573559


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   should we strip vended secrets from serialized table blob ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3575421530


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {

Review Comment:
   Will handle in https://issues.apache.org/jira/browse/HIVE-29727
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4804552240

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3561076340


##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5580,6 +5580,7 @@ public static enum ConfVars {
         + ",fs.s3a.access.key"
         + ",fs.s3a.secret.key"
         + ",fs.s3a.proxy.password"
+        + ",iceberg.mr.vended.storage.credentials"

Review Comment:
   not needed, removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3575426072


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";
+  public static final String SECRET_ACCESS_KEY = "s3.secret-access-key";
+  public static final String SESSION_TOKEN = "s3.session-token";
+  public static final String ENDPOINT = "s3.endpoint";
+  public static final String PATH_STYLE_ACCESS = "s3.path-style-access";
+
+  private static final ImmutableSet<String> SECRET_ICEBERG_KEYS = ImmutableSet.of(
+      ACCESS_KEY_ID, SECRET_ACCESS_KEY, SESSION_TOKEN);
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.VENDED_STORAGE_CREDENTIALS,
+          SerializationUtil.serializeToBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,
+      Map<String, String> jobSecrets) {
+    if (bucket != null) {
+      String s3aSecretKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aSecretKey != null) {
+        jobSecrets.put(s3aSecretKey, value);
+      }
+    }
+  }
+
+  /**
+   * Applies vended credentials to the table FileIO, merging session/catalog conf overrides (e.g. S3 endpoint).
+   * Used on executors after deserialization and on HS2 commit when the table is taken from query state.
+   */
+  public static void applyFromJobConf(Table table, Configuration conf) {
+    if (table == null || conf == null) {
+      return;
+    }
+
+    String catalogName = conf.get(InputFormatConfig.CATALOG_NAME);
+    if (shouldSkipApplyFromJobConf(catalogName, conf)) {
+      return;
+    }
+
+    FileIO io = table.io();
+    if (!(io instanceof SupportsStorageCredentials credentialIo)) {
+      return;
+    }
+
+    List<StorageCredential> credentials = resolveCredentialsForApply(table, credentialIo, conf);
+    if (credentials != null && !credentials.isEmpty()) {
+      credentialIo.setCredentials(withConfigurationOverrides(catalogName, credentials, conf));
+    }
+  }
+
+  /**
+   * Returns true when the job carries no serialized vended credentials and the catalog is not
+   * configured for credential vending (no {@code vended-credentials} REST delegation header).
+   * Otherwise apply may restore credentials from the job conf or from the table FileIO.
+   */
+  private static boolean shouldSkipApplyFromJobConf(String catalogName, Configuration conf) {
+    return StringUtils.isBlank(conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)) &&
+        !IcebergCatalogProperties.requestsVendedCredentials(catalogName, conf);
+  }
+
+  /**
+   * Chooses which vended credentials {@link #applyFromJobConf} should install on the FileIO.
+   *
+   * <p>Uses the first non-empty source: credentials already on the FileIO (typical on HS2 after
+   * table load), else the base64 list from {@link InputFormatConfig#VENDED_STORAGE_CREDENTIALS} on
+   * the task {@code conf} (propagated at plan time), else {@link #extractCredentials(Table)} from
+   * the table (including FileIO property fallbacks).
+   */
+  private static List<StorageCredential> resolveCredentialsForApply(
+      Table table, SupportsStorageCredentials credentialIo, Configuration conf) {
+
+    List<StorageCredential> credentials = credentialIo.credentials();
+    if (CollectionUtils.isEmpty(credentials)) {
+      String serialized = conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS);
+      if (StringUtils.isNotBlank(serialized)) {
+        @SuppressWarnings("unchecked")
+        List<StorageCredential> deserialized =
+            SerializationUtil.deserializeFromBase64(serialized);
+        credentials = deserialized;
+      }
+    }
+
+    if (credentials == null || credentials.isEmpty()) {
+      credentials = extractCredentials(table);
+    }
+    return credentials;
+  }
+
+  /**
+   * Returns true when the table catalog is configured for REST vended storage credentials.
+   */
+  static boolean requestsVendedCredentials(Properties properties, Configuration configuration) {
+    if (properties == null) {
+      return false;
+    }
+    return IcebergCatalogProperties.requestsVendedCredentials(
+        properties.getProperty(InputFormatConfig.CATALOG_NAME), configuration);
+  }
+
+  /**
+   * Loads a table and, if needed, bypasses the query-level cache so REST vended credentials are present on the FileIO.
+   * When vended credentials are not requested for the catalog, returns the cached table without an extra load.
+   */
+  static Table getTableWithVendedCredentials(Properties properties, Configuration configuration) {
+    Table table = IcebergTableUtil.getTable(configuration, properties);
+    if (requestsVendedCredentials(properties, configuration) && extractCredentials(table).isEmpty()) {
+      table = IcebergTableUtil.getTable(configuration, properties, true);
+    }
+    return table;
+  }
+
+  /**
+   * Reloads vended credentials at job launch when compile-time propagation missed them.
+   * Non-secret config is written to {@code jobConf}; secrets are merged into {@code TableDesc#getJobSecrets()}
+   * for {@link org.apache.hadoop.hive.ql.plan.PlanUtils#configureJobConf} (HIVE-20651).
+   */
+  static void refreshVendedCredentialsIfMissing(TableDesc tableDesc, JobConf jobConf, Configuration configuration) {
+    if (tableDesc == null || tableDesc.getProperties() == null ||
+        hasSerializedCredentials(tableDesc.getJobSecrets()) ||
+        StringUtils.isNotBlank(jobConf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))) {
+      return;
+    }
+
+    Properties props = tableDesc.getProperties();
+    if (!requestsVendedCredentials(props, configuration)) {
+      return;
+    }
+
+    String catalogName = props.getProperty(InputFormatConfig.CATALOG_NAME);
+    if (catalogName == null) {
+      return;
+    }
+
+    try {
+      Table table = getTableWithVendedCredentials(props, configuration);
+      Map<String, String> jobProps = new LinkedHashMap<>();
+      Map<String, String> secrets = new LinkedHashMap<>();
+      propagateToJob(table, catalogName, jobProps, secrets, configuration);
+      jobProps.forEach(jobConf::set);
+      mergeJobSecrets(tableDesc, secrets);
+    } catch (NoSuchTableException ex) {
+      // Table may not exist yet for CTAS; credentials will not be available.
+    }
+  }
+
+  /** Copies {@code jobSecrets} onto {@code jobConf} so the Tez driver can use S3A before tasks start. */
+  static void applyJobSecretsToJobConf(TableDesc tableDesc, JobConf jobConf) {
+    Map<String, String> secrets = tableDesc.getJobSecrets();
+    if (secrets != null) {
+      secrets.forEach(jobConf::set);
+    }
+  }
+
+  private static boolean hasSerializedCredentials(Map<String, String> jobSecrets) {
+    return jobSecrets != null &&
+        StringUtils.isNotBlank(jobSecrets.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS));
+  }
+
+  private static void mergeJobSecrets(TableDesc tableDesc, Map<String, String> secrets) {
+    if (secrets.isEmpty()) {
+      return;
+    }
+    Map<String, String> existing = tableDesc.getJobSecrets();
+    if (existing == null) {
+      tableDesc.setJobSecrets(new LinkedHashMap<>(secrets));
+    } else {
+      secrets.forEach(existing::putIfAbsent);
+    }
+  }
+
+  private static boolean isSecretKey(String icebergKey) {
+    return SECRET_ICEBERG_KEYS.contains(icebergKey);
+  }
+
+  static List<StorageCredential> extractCredentials(Table table) {
+    if (table == null) {
+      return List.of();
+    }
+    FileIO io = table.io();
+    if (io instanceof SupportsStorageCredentials credentialIo) {
+      List<StorageCredential> credentials = credentialIo.credentials();
+      if (credentials != null && !credentials.isEmpty()) {
+        return credentials;
+      }
+    }
+    return credentialsFromFileIoProperties(table, io);
+  }
+
+  private static List<StorageCredential> credentialsFromFileIoProperties(Table table, FileIO io) {
+    Map<String, String> props = io.properties();
+    if (props == null || StringUtils.isBlank(props.get(ACCESS_KEY_ID)) ||
+        StringUtils.isBlank(props.get(SECRET_ACCESS_KEY))) {
+      return List.of();
+    }
+    Map<String, String> config = new LinkedHashMap<>();
+    putIfPresent(config, props, ACCESS_KEY_ID);
+    putIfPresent(config, props, SECRET_ACCESS_KEY);
+    putIfPresent(config, props, SESSION_TOKEN);
+    putIfPresent(config, props, ENDPOINT);
+    putIfPresent(config, props, PATH_STYLE_ACCESS);
+    putIfPresent(config, props, "client.region");

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3580737768


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,499 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Base64;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConfUtil;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.BaseMetadataTable;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.aws.AwsClientProperties;
+import org.apache.iceberg.aws.s3.S3FileIOProperties;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ *
+ * <p>Limitation: credentials are minted at compile/job launch and are not refreshed — work that
+ * outlives the vended token lifetime fails authentication.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table; its catalog-qualified name keys the credentials blob,
+   *        since REST catalogs mint credentials per table (the table pointer carries the same
+   *        name, so executors resolve the matching key)
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.vendedCredentialsKey(table.name()),
+          serializeToSingleLineBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey, conf)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey, conf)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob.
+   * First table wins on a shared bucket, matching the non-secret entries, so an endpoint and its
+   * key always come from the same credential. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,

Review Comment:
   seems format specific wonder if we should refactor? how do we even know what type of storage vended credentials are for? 
   is it possible to have some generic abstraction?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct — vended credentials in `table.io().properties()` survive ser/de to executors. We still need to extract those credentials and stage them in Hive job configuration for two reasons:
   
   1. **Hadoop S3A.** Iceberg FileIO and Hadoop S3A do not share state; S3A only reads Configuration, and Tez/LLAP use S3A for `s3://` paths. At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. **Iceberg FileIO endpoint override.** `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed when REST catalogs vend S3 connection settings for their own network while Hive runs elsewhere. In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint`=`http://minio:9000`, which tasks on the host cannot reach; session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes vended credentials (from job conf or `FileIO.properties()`), replaces endpoint and path-style with session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still had `http://minio:9000`, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4568262296

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [33 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559310656


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   doesn't this copy all jobSecrets — the base64 credential blob plus fs.s3a.bucket.<bucket>.access.key/secret.key/session.token — into the plain JobConf ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4937065595

   Unlike other catalogs, the REST catalog mints credentials at the table level for the client to use in subsequent operations.                                                                                   
   I don't see this PR handles multi-table queries, last table wins on the single blob key?     


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `iceberg-rest-catalog-client` and move it there including things like HiveRESTCatalogClient, VendedCredentials, etc ?
   
   we can handle refactor in another PR, meanwhile could you please move `RestAccessDelegationMode` into the existing `org.apache.iceberg.hive.client` package next to the client



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct - vended credentials in `table.io().properties()` survive ser/de to executors. But I believe we still need parsing these credentials and moving them to Hive configuration for the following 2 reasons:
   
   We still need to put credentials in JobConf because Iceberg FileIO and Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use S3A for many s3:// paths.
   
   `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed because there can be setups where REST catalogs vend S3 connection settings for their own environment while Hive is running in a different network. For example, In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach. Hive session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes the vended credentials, replaces only the endpoint and path-style fields with the session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still contained `http://minio:9000`, and need to be overridden, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct - vended credentials in `table.io().properties()` survive ser/de to executors. But I believe we still need parsing these credentials and moving them to Hive configuration for the following 2 reasons:
   
   1. We still need to put credentials in JobConf because Iceberg FileIO and Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use S3A for many s3:// paths. 
   
   At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed because there can be setups where REST catalogs vend S3 connection settings for their own environment while Hive is running in a different network. For example, In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach. Hive session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes the vended credentials, replaces only the endpoint and path-style fields with the session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still contained `http://minio:9000`, and need to be overridden, so this step is required even when vended keys survive ser/de.



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct - vended credentials in `table.io().properties()` survive ser/de to executors. But I believe we still need parsing these credentials and moving them to Hive configuration for the following 2 reasons:
   
   1. We still need to put credentials in JobConf because Iceberg FileIO and Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use S3A for many s3:// paths. At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed because there can be setups where REST catalogs vend S3 connection settings for their own environment while Hive is running in a different network. For example, In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach. Hive session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes the vended credentials, replaces only the endpoint and path-style fields with the session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still contained `http://minio:9000`, and need to be overridden, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3581426435


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,499 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Base64;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConfUtil;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.BaseMetadataTable;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.aws.AwsClientProperties;
+import org.apache.iceberg.aws.s3.S3FileIOProperties;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ *
+ * <p>Limitation: credentials are minted at compile/job launch and are not refreshed — work that
+ * outlives the vended token lifetime fails authentication.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table; its catalog-qualified name keys the credentials blob,
+   *        since REST catalogs mint credentials per table (the table pointer carries the same
+   *        name, so executors resolve the matching key)
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.vendedCredentialsKey(table.name()),
+          serializeToSingleLineBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey, conf)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey, conf)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob.
+   * First table wins on a shared bucket, matching the non-secret entries, so an endpoint and its
+   * key always come from the same credential. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,

Review Comment:
   Agreed. Iceberg credential transport is already generic, we serialize/restore the `StorageCredential` list for Iceberg `FileIO` on executors. The S3-specific part is the Hadoop propagation (`toS3aBucketProperty()` / `ddSecretCredentialEntry()`), which maps vended config to `fs.s3a.bucket.*` because Tez/LLAP still use Hadoop FileSystem for some `s3://` paths.
   
   This PR intentionally keeps that scope to S3 (Gravitino q-test + S3FileIO). Can we handle provider-aware propagation for GCS/ADLS/OSS in a follow-up, deriving storage type from `StorageCredential.prefix()` and mapping to the right Hadoop FS / Iceberg IO settings per provider?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4984642191

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4627712278

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (`SerializableTable`) into `JobConf` for Tez/LLAP, and vended credentials on `FileIO` typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST loadTable, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + S3A bucket keys) and restore them in deserializeTable via applyFromJobConf.
   
   The main place we mutate vended credential content is `withConfigurationOverrides()` method. REST catalogs can vend connectivity settings from their network view (e.g. `http://minio:9000` when the catalog runs in Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint=http://host:9000`). That method overrides only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on `FileIO` from loadTable).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3580616276


##########
itests/qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/OzoneS3GatewayContainers.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.containers.wait.strategy.WaitAllStrategy;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Minimal Apache Ozone cluster with S3 Gateway for q-tests that need an S3-compatible endpoint.
+ *
+ * <p>Layout matches {@code packaging/src/docker/storage/ozone/docker-compose.yml}: scm, om, datanode,
+ * and s3g on a shared Docker network. S3G is reachable as {@link #S3_DOCKER_ALIAS}:{@link #S3G_PORT}
+ * from other containers (for example Gravitino) and via the published host port from the test JVM.
+ */
+public final class OzoneS3GatewayContainers {

Review Comment:
   should we move this to hive-it-util?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3577296875


##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   ````
   Utilities.copyTableJobPropertiesToConf(currDesc.getTableDesc(), job);
   
     if (currDesc.getTableDesc().getJobSecrets() != null) {
       currDesc.getTableDesc().getJobSecrets().forEach(job::set);
     }
   ````



##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   ````
   Utilities.copyTableJobPropertiesToConf(currDesc.getTableDesc(), job);
   
   if (currDesc.getTableDesc().getJobSecrets() != null) {
     currDesc.getTableDesc().getJobSecrets().forEach(job::set);
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3574503368


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:
##########
@@ -861,36 +861,44 @@ private static void configureJobPropertiesForStorageHandler(boolean input,
       if (storageHandler != null) {
         Map<String, String> jobProperties = new LinkedHashMap<>();
         Map<String, String> jobSecrets = new LinkedHashMap<>();
-        if(input) {
-            try {
-                storageHandler.configureInputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureInputJobProperties not found "+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
-
-            try{
-              storageHandler.configureInputJobCredentials(
-                tableDesc,
-                jobSecrets);
-            } catch(AbstractMethodError e) {
-              // ignore
-              LOG.info("configureInputJobSecrets not found");
-            }
+        if (input) {
+          try {
+            storageHandler.configureInputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureInputJobProperties not found "+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(
+              tableDesc,
+              jobSecrets);
+          } catch (AbstractMethodError e) {
+            // ignore
+            LOG.info("configureInputJobSecrets not found");
+          }
         }
         else {
-            try {
-                storageHandler.configureOutputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureOutputJobProperties not found"+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
+          try {
+            storageHandler.configureOutputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureOutputJobProperties not found"+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(

Review Comment:
   Fixed by the 'Strip → pass → restore' patch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559693131


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:
##########
@@ -861,36 +861,44 @@ private static void configureJobPropertiesForStorageHandler(boolean input,
       if (storageHandler != null) {
         Map<String, String> jobProperties = new LinkedHashMap<>();
         Map<String, String> jobSecrets = new LinkedHashMap<>();
-        if(input) {
-            try {
-                storageHandler.configureInputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureInputJobProperties not found "+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
-
-            try{
-              storageHandler.configureInputJobCredentials(
-                tableDesc,
-                jobSecrets);
-            } catch(AbstractMethodError e) {
-              // ignore
-              LOG.info("configureInputJobSecrets not found");
-            }
+        if (input) {
+          try {
+            storageHandler.configureInputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureInputJobProperties not found "+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(
+              tableDesc,
+              jobSecrets);
+          } catch (AbstractMethodError e) {
+            // ignore
+            LOG.info("configureInputJobSecrets not found");
+          }
         }
         else {
-            try {
-                storageHandler.configureOutputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureOutputJobProperties not found"+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
+          try {
+            storageHandler.configureOutputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureOutputJobProperties not found"+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(

Review Comment:
   is that really needed? 
   
   can we move it to FileSinkOperator?
   
   ````
     private void copyTableSecretsAndJobPropertiesToConf(JobConf job) throws HiveException {
       try {
         Utilities.copyJobSecretToTableProperties(conf.getTableInfo());
       } catch (IOException e) {
         throw new HiveException(e);
       }
       Utilities.copyTableJobPropertiesToConf(conf.getTableInfo(), job);
     }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559310656


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   doesn't this copy the whole secrets map — raw fs.s3a.bucket.<b>.access.key/secret.key/session.token plus the base64 credential blob — into the plain JobCon ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559363612


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   In Hive secrets ride in Hadoop Credentials (PlanUtils.configureJobConf → jobConf.getCredentials().addSecretKey(...)) and are restored on tasks by Utilities.copyJobSecretToTableProperties



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559614815


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1681,7 +1705,11 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
     PartitionSpec spec;
     String bytes;
     try {
-      Table table = IcebergTableUtil.getTable(configuration, props);
+      boolean isVendedCredentials =
+          IcebergVendedCredentialUtil.requestsVendedCredentials(props, configuration);
+      Table table = isVendedCredentials ?

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls `HiveTableUtil.serializeTable(table, ...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct — vended credentials in `table.io().properties()` survive ser/de to executors. We still zneed to extract those credentials and stage them in Hive job configuration for two reasons:
   
   1. **Hadoop S3A.** Iceberg FileIO and Hadoop S3A do not share state; S3A only reads Configuration, and Tez/LLAP use S3A for `s3://` paths. At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. Iceberg FileIO endpoint override. `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed when REST catalogs vend S3 connection settings for their own network while Hive runs elsewhere. In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint`=`http://minio:9000`, which tasks on the host cannot reach; session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes vended credentials (from job conf or `FileIO.properties()`), replaces only endpoint and path-style with session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still had `http://minio:9000`, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (SerializableTable) into JobConf for Tez/LLAP, and vended credentials on FileIO typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST `loadTable`, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + `S3A` bucket keys) and restore them in `deserializeTable` via `applyFromJobConf`.
   
   On "store as-is / restore as-is" vs the helper methods:
   
   **`resolveCredentialsForApply()`**: At apply time credentials can live in different places depending on context:
   
   - Tez/LLAP tasks: deserialized Table has an empty `FileIO.credentials()`; the source of truth is the serialized blob in task conf (see `applyFromJobConfRestoresCredentialsOnExecutor`).
   - HS2 commit: the table often still has credentials from the earlier loadTable in QueryState; we prefer those over re-reading the blob.
   - Fallback: extractCredentials(table) if both are empty.
   Without this resolution logic we'd need separate executor vs commit paths anyway.
   
   **`credentialsFromFileIoProperties()`**: needed at plan/propagate time. Some REST/FileIO implementations populate `s3.access-key-id` / `s3.secret-access-key` in `FileIO.properties()` while credentials() is still empty after loadTable. extractCredentials uses this fallback so propagateToJob doesn't skip vending even though the table load succeeded (extractCredentialsFromFileIoPropertiesWhenCredentialListEmpty). It also backs the last-resort branch in resolveCredentialsForApply.
   
   **`withConfigurationOverrides`**: this is the part that does mutate vended credentials. REST catalogs often vend connectivity settings from their network view (e.g. `http://minio:9000` inside Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint`=`http://host:9000`). We override only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on FileIO from `loadTable`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (SerializableTable) into JobConf for Tez/LLAP, and vended credentials on FileIO typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST `loadTable`, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + `S3A` bucket keys) and restore them in `deserializeTable` via `applyFromJobConf`.
   
   On "store as-is / restore as-is" vs the helper methods:
   
   **`resolveCredentialsForApply()`**: At apply time credentials can live in different places depending on context:
   
   - Tez/LLAP tasks: deserialized Table has an empty `FileIO.credentials()`; the source of truth is the serialized blob in task conf (see `applyFromJobConfRestoresCredentialsOnExecutor`).
   - HS2 commit: the table often still has credentials from the earlier loadTable in QueryState; we prefer those over re-reading the blob.
   - Fallback: extractCredentials(table) if both are empty.
   
   **`credentialsFromFileIoProperties()`**: needed at plan/propagate time. Some REST/FileIO implementations populate `s3.access-key-id` / `s3.secret-access-key` in `FileIO.properties()` while credentials() is still empty after loadTable. extractCredentials uses this fallback so propagateToJob doesn't skip vending even though the table load succeeded (extractCredentialsFromFileIoPropertiesWhenCredentialListEmpty). It also backs the last-resort branch in resolveCredentialsForApply.
   
   **`withConfigurationOverrides`**: this is the part that does mutate vended credentials. REST catalogs often vend connectivity settings from their network view (e.g. `http://minio:9000` inside Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint`=`http://host:9000`). We override only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on FileIO from `loadTable`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3575415136


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java:
##########
@@ -80,6 +80,12 @@ private InputFormatConfig() {
 
   public static final String CATALOG_CONFIG_PREFIX = "iceberg.catalog.";
 
+  /**
+   * Base64-serialized list of {@link org.apache.iceberg.io.StorageCredential} for Tez/LLAP executors.
+   * Stored in {@code TableDesc#jobSecrets} (HIVE-20651), not in job properties.
+   */
+  public static final String VENDED_STORAGE_CREDENTIALS = "iceberg.mr.vended.storage.credentials";

Review Comment:
   Handled in the patch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4961423381

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3575417602


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559158793


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/IcebergCatalogProperties.java:
##########
@@ -104,6 +105,19 @@ public static String catalogPropertyConfigKey(String catalogName, String catalog
     return String.format("%s%s.%s", CATALOG_CONFIG_PREFIX, catalogName, catalogProperty);
   }
 
+  /**
+   * Returns true when the catalog is configured to request REST vended storage credentials via
+   * {@link #REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+   */
+  public static boolean requestsVendedCredentials(String catalogName, Configuration conf) {

Review Comment:
   maybe could be also moved in IcebergVendedCredentialUtil ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there including things like HiveRESTCatalogClient, VendedCredentials, etc ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559636516


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -284,6 +284,21 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
     return null;
   }
 
+  @Override
+  public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {

Review Comment:
   what about `configureOutputJobCredentials` (for output tables) ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3580381650


##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4967113181

   I think we are also missing executor-side credential refresh. In Trino AbstractIcebergRestVendedCredentialsProvider re-vends near expiry via client.refresh-credentials-endpoint


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3581391921


##########
itests/qtest-iceberg/src/test/java/org/apache/hadoop/hive/cli/OzoneS3GatewayContainers.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.cli;
+
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.containers.wait.strategy.WaitAllStrategy;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Minimal Apache Ozone cluster with S3 Gateway for q-tests that need an S3-compatible endpoint.
+ *
+ * <p>Layout matches {@code packaging/src/docker/storage/ozone/docker-compose.yml}: scm, om, datanode,
+ * and s3g on a shared Docker network. S3G is reachable as {@link #S3_DOCKER_ALIAS}:{@link #S3G_PORT}
+ * from other containers (for example Gravitino) and via the published host port from the test JVM.
+ */
+public final class OzoneS3GatewayContainers {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3578697912


##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   ````
   Subject: [PATCH] new patch
   ---
   Index: iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java
   IDEA additional info:
   Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
   <+>UTF-8
   ===================================================================
   diff --git a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java
   --- a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java	(revision 4c3214ed63152c640e1febbd46873016775a3bdc)
   +++ b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java	(date 1784029753143)
   @@ -26,6 +26,7 @@
    import java.util.Properties;
    import org.apache.commons.lang3.StringUtils;
    import org.apache.hadoop.conf.Configuration;
   +import org.apache.hadoop.hive.conf.HiveConfUtil;
    import org.apache.hadoop.hive.ql.plan.TableDesc;
    import org.apache.hadoop.mapred.JobConf;
    import org.apache.iceberg.BaseMetadataTable;
   @@ -46,7 +47,6 @@
    import org.apache.iceberg.io.SupportsStorageCredentials;
    import org.apache.iceberg.mr.InputFormatConfig;
    import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
   -import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
    import org.apache.iceberg.relocated.com.google.common.collect.Lists;
    import org.apache.iceberg.util.SerializationUtil;
    
   @@ -59,10 +59,6 @@
     */
    public final class IcebergVendedCredentialUtil {
    
   -  /** Vended config safe to display; any key not listed here routes to the secrets channel. */
   -  private static final ImmutableSet<String> NON_SECRET_ICEBERG_KEYS = ImmutableSet.of(
   -      S3FileIOProperties.ENDPOINT, S3FileIOProperties.PATH_STYLE_ACCESS, AwsClientProperties.CLIENT_REGION);
   -
      private IcebergVendedCredentialUtil() {
      }
    
   @@ -135,11 +131,11 @@
        }
        String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
    
   -    if (jobProperties != null && !isSecretKey(icebergKey)) {
   +    if (jobProperties != null && !isSecretKey(icebergKey, conf)) {
          addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
        }
    
   -    if (jobSecrets != null && isSecretKey(icebergKey)) {
   +    if (jobSecrets != null && isSecretKey(icebergKey, conf)) {
          addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
        }
      }
   @@ -320,13 +316,11 @@
    
        FileIO io = table.io();
        Map<String, String> ioProps = new LinkedHashMap<>();
   -    if (io.properties() != null) {
   -      io.properties().forEach((key, value) -> {
   -        if (NON_SECRET_ICEBERG_KEYS.contains(key)) {
   -          ioProps.put(key, value);
   -        }
   -      });
   -    }
   +    io.properties().forEach((k, v) -> {
   +      if (!isSecretKey(k, conf)) {
   +        ioProps.put(k, v);
   +      }
   +    });
        FileIO cleanIo = CatalogUtil.loadFileIO(io.getClass().getName(), ioProps, conf);
    
        return new BaseTable(
   @@ -370,8 +364,14 @@
        }
      }
    
   -  private static boolean isSecretKey(String icebergKey) {
   -    return !NON_SECRET_ICEBERG_KEYS.contains(icebergKey);
   +  /**
   +   * A vended config key is secret — routed to the Credentials channel, never job properties, and
   +   * stripped from the serialized table's FileIO — exactly when {@code hive.conf.hidden.list} covers
   +   * it. That is Hive's own registry of sensitive configuration (the same one behind
   +   * {@code HiveConf#isHiddenConfig}) that masks values in EXPLAIN, the Tez UI, and ATS.
   +   */
   +  private static boolean isSecretKey(String key, Configuration conf) {
   +    return HiveConfUtil.getHiddenSet(conf).stream().anyMatch(key::startsWith);
      }
    
      static List<StorageCredential> extractCredentials(Table table) {
   Index: ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
   IDEA additional info:
   Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
   <+>UTF-8
   ===================================================================
   diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
   --- a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java	(revision 4c3214ed63152c640e1febbd46873016775a3bdc)
   +++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java	(date 1784029871481)
   @@ -375,6 +375,13 @@
    
          Class<? extends InputFormat> formatter = currDesc.getInputFileFormatClass();
          Utilities.copyTableJobPropertiesToConf(currDesc.getTableDesc(), job);
   +      // A fetch task runs in-process without a DAG, so storage-handler secrets (HIVE-20651) are
   +      // never restored from the job Credentials as on a task. Surface them into this process-local
   +      // conf (never serialized or shown in EXPLAIN) so in-process readers can access storage.
   +      Map<String, String> jobSecrets = currDesc.getTableDesc().getJobSecrets();
   +      if (jobSecrets != null) {
   +        jobSecrets.forEach(job::set);
   +      }
          InputFormat inputFormat = getInputFormatFromCache(formatter, job);
          if(inputFormat instanceof HiveSequenceFileInputFormat) {
            // input format could be cached, in which case we need to reset the list of files to fetch
   Index: iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q
   IDEA additional info:
   Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
   <+>UTF-8
   ===================================================================
   diff --git a/iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q b/iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q
   --- a/iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q	(revision 4c3214ed63152c640e1febbd46873016775a3bdc)
   +++ b/iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q	(date 1784016928801)
   @@ -29,9 +29,6 @@
    set metastore.catalog.default=ice01;
    set iceberg.catalog.ice01.type=rest;
    set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
   ---! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
   ---! restore them, so fetch conversion must be off for credential-vending catalogs.
   -set hive.fetch.task.conversion=none;
    
    --! REST URI, OAuth, MinIO + Gravitino S3 warehouse / credential vending, and host S3A are set in
    --! TestIcebergRESTCatalogGravitinoLlapLocalCliDriver.
   Index: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
   IDEA additional info:
   Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
   <+>UTF-8
   ===================================================================
   diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
   --- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	(revision 4c3214ed63152c640e1febbd46873016775a3bdc)
   +++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	(date 1784021299978)
   @@ -5601,6 +5601,10 @@
            + ",fs.s3a.secret.key"
            + ",fs.s3a.proxy.password"
            + ",iceberg.vended.storage.credentials"
   +        // Iceberg FileIO vended credential keys (S3FileIOProperties)
   +        + ",s3.access-key-id"
   +        + ",s3.secret-access-key"
   +        + ",s3.session-token"
            + ",dfs.adls.oauth2.credential"
            + ",fs.adl.oauth2.credential"
            + ",fs.azure.account.oauth2.client.secret"
   
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin merged PR #6474:
URL: https://github.com/apache/hive/pull/6474


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct - vended credentials in `table.io().properties()` survive ser/de to executors.
   
   We still need to put credentials in JobConf because Iceberg FileIO and Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use S3A for many s3:// paths.
   
   `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed because there can be setups where REST catalogs vend S3 connection settings for their own environment while Hive is running in a different network. For example, In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach. Hive session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes the vended credentials, replaces only the endpoint and path-style fields with the session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still contained `http://minio:9000`, and need to be overridden, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4846035120

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559693131


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:
##########
@@ -861,36 +861,44 @@ private static void configureJobPropertiesForStorageHandler(boolean input,
       if (storageHandler != null) {
         Map<String, String> jobProperties = new LinkedHashMap<>();
         Map<String, String> jobSecrets = new LinkedHashMap<>();
-        if(input) {
-            try {
-                storageHandler.configureInputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureInputJobProperties not found "+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
-
-            try{
-              storageHandler.configureInputJobCredentials(
-                tableDesc,
-                jobSecrets);
-            } catch(AbstractMethodError e) {
-              // ignore
-              LOG.info("configureInputJobSecrets not found");
-            }
+        if (input) {
+          try {
+            storageHandler.configureInputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureInputJobProperties not found "+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(
+              tableDesc,
+              jobSecrets);
+          } catch (AbstractMethodError e) {
+            // ignore
+            LOG.info("configureInputJobSecrets not found");
+          }
         }
         else {
-            try {
-                storageHandler.configureOutputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureOutputJobProperties not found"+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
+          try {
+            storageHandler.configureOutputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureOutputJobProperties not found"+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(

Review Comment:
   is that really needed? 
   
   can we do Utilities.copyJobSecretToTableProperties(conf.getTableInfo()) in FileSinkOperator.initializeOp ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3561118063


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   Create a new ticket https://issues.apache.org/jira/browse/HIVE-29727 for the refactor.
   
   Moved `RestAccessDelegationMode` to `org.apache.iceberg.hive.client`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3560549527


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";
+  public static final String SECRET_ACCESS_KEY = "s3.secret-access-key";
+  public static final String SESSION_TOKEN = "s3.session-token";
+  public static final String ENDPOINT = "s3.endpoint";
+  public static final String PATH_STYLE_ACCESS = "s3.path-style-access";
+
+  private static final ImmutableSet<String> SECRET_ICEBERG_KEYS = ImmutableSet.of(
+      ACCESS_KEY_ID, SECRET_ACCESS_KEY, SESSION_TOKEN);
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.VENDED_STORAGE_CREDENTIALS,
+          SerializationUtil.serializeToBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,
+      Map<String, String> jobSecrets) {
+    if (bucket != null) {
+      String s3aSecretKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aSecretKey != null) {
+        jobSecrets.put(s3aSecretKey, value);
+      }
+    }
+  }
+
+  /**
+   * Applies vended credentials to the table FileIO, merging session/catalog conf overrides (e.g. S3 endpoint).
+   * Used on executors after deserialization and on HS2 commit when the table is taken from query state.
+   */
+  public static void applyFromJobConf(Table table, Configuration conf) {
+    if (table == null || conf == null) {
+      return;
+    }
+
+    String catalogName = conf.get(InputFormatConfig.CATALOG_NAME);
+    if (shouldSkipApplyFromJobConf(catalogName, conf)) {
+      return;
+    }
+
+    FileIO io = table.io();
+    if (!(io instanceof SupportsStorageCredentials credentialIo)) {
+      return;
+    }
+
+    List<StorageCredential> credentials = resolveCredentialsForApply(table, credentialIo, conf);
+    if (credentials != null && !credentials.isEmpty()) {
+      credentialIo.setCredentials(withConfigurationOverrides(catalogName, credentials, conf));
+    }
+  }
+
+  /**
+   * Returns true when the job carries no serialized vended credentials and the catalog is not
+   * configured for credential vending (no {@code vended-credentials} REST delegation header).
+   * Otherwise apply may restore credentials from the job conf or from the table FileIO.
+   */
+  private static boolean shouldSkipApplyFromJobConf(String catalogName, Configuration conf) {
+    return StringUtils.isBlank(conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)) &&
+        !IcebergCatalogProperties.requestsVendedCredentials(catalogName, conf);
+  }
+
+  /**
+   * Chooses which vended credentials {@link #applyFromJobConf} should install on the FileIO.
+   *
+   * <p>Uses the first non-empty source: credentials already on the FileIO (typical on HS2 after
+   * table load), else the base64 list from {@link InputFormatConfig#VENDED_STORAGE_CREDENTIALS} on
+   * the task {@code conf} (propagated at plan time), else {@link #extractCredentials(Table)} from
+   * the table (including FileIO property fallbacks).
+   */
+  private static List<StorageCredential> resolveCredentialsForApply(
+      Table table, SupportsStorageCredentials credentialIo, Configuration conf) {
+
+    List<StorageCredential> credentials = credentialIo.credentials();
+    if (CollectionUtils.isEmpty(credentials)) {
+      String serialized = conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS);
+      if (StringUtils.isNotBlank(serialized)) {
+        @SuppressWarnings("unchecked")
+        List<StorageCredential> deserialized =
+            SerializationUtil.deserializeFromBase64(serialized);
+        credentials = deserialized;
+      }
+    }
+
+    if (credentials == null || credentials.isEmpty()) {
+      credentials = extractCredentials(table);
+    }
+    return credentials;
+  }
+
+  /**
+   * Returns true when the table catalog is configured for REST vended storage credentials.
+   */
+  static boolean requestsVendedCredentials(Properties properties, Configuration configuration) {
+    if (properties == null) {
+      return false;
+    }
+    return IcebergCatalogProperties.requestsVendedCredentials(
+        properties.getProperty(InputFormatConfig.CATALOG_NAME), configuration);
+  }
+
+  /**
+   * Loads a table and, if needed, bypasses the query-level cache so REST vended credentials are present on the FileIO.
+   * When vended credentials are not requested for the catalog, returns the cached table without an extra load.
+   */
+  static Table getTableWithVendedCredentials(Properties properties, Configuration configuration) {
+    Table table = IcebergTableUtil.getTable(configuration, properties);
+    if (requestsVendedCredentials(properties, configuration) && extractCredentials(table).isEmpty()) {
+      table = IcebergTableUtil.getTable(configuration, properties, true);
+    }
+    return table;
+  }
+
+  /**
+   * Reloads vended credentials at job launch when compile-time propagation missed them.
+   * Non-secret config is written to {@code jobConf}; secrets are merged into {@code TableDesc#getJobSecrets()}
+   * for {@link org.apache.hadoop.hive.ql.plan.PlanUtils#configureJobConf} (HIVE-20651).
+   */
+  static void refreshVendedCredentialsIfMissing(TableDesc tableDesc, JobConf jobConf, Configuration configuration) {
+    if (tableDesc == null || tableDesc.getProperties() == null ||
+        hasSerializedCredentials(tableDesc.getJobSecrets()) ||
+        StringUtils.isNotBlank(jobConf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))) {
+      return;
+    }
+
+    Properties props = tableDesc.getProperties();
+    if (!requestsVendedCredentials(props, configuration)) {
+      return;
+    }
+
+    String catalogName = props.getProperty(InputFormatConfig.CATALOG_NAME);
+    if (catalogName == null) {
+      return;
+    }
+
+    try {
+      Table table = getTableWithVendedCredentials(props, configuration);
+      Map<String, String> jobProps = new LinkedHashMap<>();
+      Map<String, String> secrets = new LinkedHashMap<>();
+      propagateToJob(table, catalogName, jobProps, secrets, configuration);
+      jobProps.forEach(jobConf::set);
+      mergeJobSecrets(tableDesc, secrets);
+    } catch (NoSuchTableException ex) {
+      // Table may not exist yet for CTAS; credentials will not be available.
+    }
+  }
+
+  /** Copies {@code jobSecrets} onto {@code jobConf} so the Tez driver can use S3A before tasks start. */
+  static void applyJobSecretsToJobConf(TableDesc tableDesc, JobConf jobConf) {
+    Map<String, String> secrets = tableDesc.getJobSecrets();
+    if (secrets != null) {
+      secrets.forEach(jobConf::set);
+    }
+  }
+
+  private static boolean hasSerializedCredentials(Map<String, String> jobSecrets) {
+    return jobSecrets != null &&
+        StringUtils.isNotBlank(jobSecrets.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS));
+  }
+
+  private static void mergeJobSecrets(TableDesc tableDesc, Map<String, String> secrets) {
+    if (secrets.isEmpty()) {
+      return;
+    }
+    Map<String, String> existing = tableDesc.getJobSecrets();
+    if (existing == null) {
+      tableDesc.setJobSecrets(new LinkedHashMap<>(secrets));
+    } else {
+      secrets.forEach(existing::putIfAbsent);
+    }
+  }
+
+  private static boolean isSecretKey(String icebergKey) {
+    return SECRET_ICEBERG_KEYS.contains(icebergKey);
+  }
+
+  static List<StorageCredential> extractCredentials(Table table) {
+    if (table == null) {
+      return List.of();
+    }
+    FileIO io = table.io();
+    if (io instanceof SupportsStorageCredentials credentialIo) {
+      List<StorageCredential> credentials = credentialIo.credentials();
+      if (credentials != null && !credentials.isEmpty()) {
+        return credentials;
+      }
+    }
+    return credentialsFromFileIoProperties(table, io);
+  }
+
+  private static List<StorageCredential> credentialsFromFileIoProperties(Table table, FileIO io) {
+    Map<String, String> props = io.properties();
+    if (props == null || StringUtils.isBlank(props.get(ACCESS_KEY_ID)) ||
+        StringUtils.isBlank(props.get(SECRET_ACCESS_KEY))) {
+      return List.of();
+    }
+    Map<String, String> config = new LinkedHashMap<>();
+    putIfPresent(config, props, ACCESS_KEY_ID);
+    putIfPresent(config, props, SECRET_ACCESS_KEY);
+    putIfPresent(config, props, SESSION_TOKEN);
+    putIfPresent(config, props, ENDPOINT);
+    putIfPresent(config, props, PATH_STYLE_ACCESS);
+    putIfPresent(config, props, "client.region");

Review Comment:
   AwsClientProperties.CLIENT_REGION



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3561236582


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/IcebergCatalogProperties.java:
##########
@@ -104,6 +105,19 @@ public static String catalogPropertyConfigKey(String catalogName, String catalog
     return String.format("%s%s.%s", CATALOG_CONFIG_PREFIX, catalogName, catalogProperty);
   }
 
+  /**
+   * Returns true when the catalog is configured to request REST vended storage credentials via
+   * {@link #REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+   */
+  public static boolean requestsVendedCredentials(String catalogName, Configuration conf) {

Review Comment:
   I think it is better to leave it in `IcebergCatalogProperties` because this this catalog configuration method and it is also called from `TestRestAccessDelegationMode` which lives in `iceberg-catalog` module.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559360752


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   In Hive secrets ride in Hadoop Credentials (PlanUtils.configureJobConf → jobConf.getCredentials().addSecretKey(...)) and are restored on tasks by Utilities.copyJobSecretToTableProperties



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   In Hive secrets ride in Hadoop Credentials (PlanUtils.configureJobConf → jobConf.getCredentials().addSecretKey(...)) and are restored on tasks by Utilities.copyJobSecretToTableProperties



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559573559


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls HiveTableUtil.serializeTable(table, ...)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3570589897


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {

Review Comment:
   also client's code, please move to appropriate package



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4915322254

   isn't MinIO is in "maintenance mode" and is no longer accepting new changes? should we use ozone instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (SerializableTable) into JobConf for Tez/LLAP, and vended credentials on FileIO typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST `loadTable`, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + `S3A` bucket keys) and restore them in `deserializeTable` via `applyFromJobConf`.
   
   There is one place where the vended credentials can be updated: `withConfigurationOverrides` method. REST catalogs can vend connectivity settings from their network view (e.g. `http://minio:9000` when REST catalog runs in Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint`=`http://host:9000`). This method overrides only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on FileIO from `loadTable`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558688889


##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5580,6 +5580,7 @@ public static enum ConfVars {
         + ",fs.s3a.access.key"
         + ",fs.s3a.secret.key"
         + ",fs.s3a.proxy.password"
+        + ",iceberg.mr.vended.storage.credentials"

Review Comment:
   does it need to have `.mr` prefix?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559573559


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls `HiveTableUtil.serializeTable(table, ...)`



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls `HiveTableUtil.serializeTable(table, ...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3560290506


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java:
##########
@@ -80,6 +80,12 @@ private InputFormatConfig() {
 
   public static final String CATALOG_CONFIG_PREFIX = "iceberg.catalog.";
 
+  /**
+   * Base64-serialized list of {@link org.apache.iceberg.io.StorageCredential} for Tez/LLAP executors.
+   * Stored in {@code TableDesc#jobSecrets} (HIVE-20651), not in job properties.
+   */
+  public static final String VENDED_STORAGE_CREDENTIALS = "iceberg.mr.vended.storage.credentials";

Review Comment:
   REST catalog mints credentials at the table level for the client. So do we need to key the credentials blob by table name —`iceberg.mr.serialized.table.<name> ?
   
   ````
   public static String vendedCredentialsKey(String tableName) {
     return VENDED_STORAGE_CREDENTIALS + "." + tableName;
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there including rest of things like VendedCredentials, client creation, etc ?
   
   alternative would be moving the enum next to the REST client in handler instead



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3560290506


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java:
##########
@@ -80,6 +80,12 @@ private InputFormatConfig() {
 
   public static final String CATALOG_CONFIG_PREFIX = "iceberg.catalog.";
 
+  /**
+   * Base64-serialized list of {@link org.apache.iceberg.io.StorageCredential} for Tez/LLAP executors.
+   * Stored in {@code TableDesc#jobSecrets} (HIVE-20651), not in job properties.
+   */
+  public static final String VENDED_STORAGE_CREDENTIALS = "iceberg.mr.vended.storage.credentials";

Review Comment:
   REST catalog mints credentials at the table level for the client. So do we need to key the credentials blob by table name — `iceberg.mr.serialized.table.<name> ?



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java:
##########
@@ -80,6 +80,12 @@ private InputFormatConfig() {
 
   public static final String CATALOG_CONFIG_PREFIX = "iceberg.catalog.";
 
+  /**
+   * Base64-serialized list of {@link org.apache.iceberg.io.StorageCredential} for Tez/LLAP executors.
+   * Stored in {@code TableDesc#jobSecrets} (HIVE-20651), not in job properties.
+   */
+  public static final String VENDED_STORAGE_CREDENTIALS = "iceberg.mr.vended.storage.credentials";

Review Comment:
   REST catalog mints credentials at the table level for the client. So do we need to key the credentials blob by table name —`iceberg.mr.serialized.table.<name> ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4970984658

   > isn't MinIO is in "maintenance mode" and is no longer accepting new changes? the image eventually could disappear from Docker Hub so i am not sure we should be adding dead project dependency. should we use ozone instead?
   
   Switched to using Ozone instead of Minio.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3575419045


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";
+  public static final String SECRET_ACCESS_KEY = "s3.secret-access-key";
+  public static final String SESSION_TOKEN = "s3.session-token";
+  public static final String ENDPOINT = "s3.endpoint";
+  public static final String PATH_STYLE_ACCESS = "s3.path-style-access";
+
+  private static final ImmutableSet<String> SECRET_ICEBERG_KEYS = ImmutableSet.of(
+      ACCESS_KEY_ID, SECRET_ACCESS_KEY, SESSION_TOKEN);
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.VENDED_STORAGE_CREDENTIALS,
+          SerializationUtil.serializeToBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,
+      Map<String, String> jobSecrets) {
+    if (bucket != null) {
+      String s3aSecretKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aSecretKey != null) {
+        jobSecrets.put(s3aSecretKey, value);
+      }
+    }
+  }
+
+  /**
+   * Applies vended credentials to the table FileIO, merging session/catalog conf overrides (e.g. S3 endpoint).
+   * Used on executors after deserialization and on HS2 commit when the table is taken from query state.
+   */
+  public static void applyFromJobConf(Table table, Configuration conf) {
+    if (table == null || conf == null) {
+      return;
+    }
+
+    String catalogName = conf.get(InputFormatConfig.CATALOG_NAME);
+    if (shouldSkipApplyFromJobConf(catalogName, conf)) {
+      return;
+    }
+
+    FileIO io = table.io();
+    if (!(io instanceof SupportsStorageCredentials credentialIo)) {
+      return;
+    }
+
+    List<StorageCredential> credentials = resolveCredentialsForApply(table, credentialIo, conf);
+    if (credentials != null && !credentials.isEmpty()) {
+      credentialIo.setCredentials(withConfigurationOverrides(catalogName, credentials, conf));
+    }
+  }
+
+  /**
+   * Returns true when the job carries no serialized vended credentials and the catalog is not
+   * configured for credential vending (no {@code vended-credentials} REST delegation header).
+   * Otherwise apply may restore credentials from the job conf or from the table FileIO.
+   */
+  private static boolean shouldSkipApplyFromJobConf(String catalogName, Configuration conf) {
+    return StringUtils.isBlank(conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)) &&
+        !IcebergCatalogProperties.requestsVendedCredentials(catalogName, conf);
+  }
+
+  /**
+   * Chooses which vended credentials {@link #applyFromJobConf} should install on the FileIO.
+   *
+   * <p>Uses the first non-empty source: credentials already on the FileIO (typical on HS2 after
+   * table load), else the base64 list from {@link InputFormatConfig#VENDED_STORAGE_CREDENTIALS} on
+   * the task {@code conf} (propagated at plan time), else {@link #extractCredentials(Table)} from
+   * the table (including FileIO property fallbacks).
+   */
+  private static List<StorageCredential> resolveCredentialsForApply(
+      Table table, SupportsStorageCredentials credentialIo, Configuration conf) {
+
+    List<StorageCredential> credentials = credentialIo.credentials();
+    if (CollectionUtils.isEmpty(credentials)) {
+      String serialized = conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS);
+      if (StringUtils.isNotBlank(serialized)) {
+        @SuppressWarnings("unchecked")
+        List<StorageCredential> deserialized =
+            SerializationUtil.deserializeFromBase64(serialized);
+        credentials = deserialized;
+      }
+    }
+
+    if (credentials == null || credentials.isEmpty()) {
+      credentials = extractCredentials(table);
+    }
+    return credentials;
+  }
+
+  /**
+   * Returns true when the table catalog is configured for REST vended storage credentials.
+   */
+  static boolean requestsVendedCredentials(Properties properties, Configuration configuration) {
+    if (properties == null) {
+      return false;
+    }
+    return IcebergCatalogProperties.requestsVendedCredentials(
+        properties.getProperty(InputFormatConfig.CATALOG_NAME), configuration);
+  }
+
+  /**
+   * Loads a table and, if needed, bypasses the query-level cache so REST vended credentials are present on the FileIO.
+   * When vended credentials are not requested for the catalog, returns the cached table without an extra load.
+   */
+  static Table getTableWithVendedCredentials(Properties properties, Configuration configuration) {
+    Table table = IcebergTableUtil.getTable(configuration, properties);
+    if (requestsVendedCredentials(properties, configuration) && extractCredentials(table).isEmpty()) {
+      table = IcebergTableUtil.getTable(configuration, properties, true);
+    }
+    return table;
+  }
+
+  /**
+   * Reloads vended credentials at job launch when compile-time propagation missed them.
+   * Non-secret config is written to {@code jobConf}; secrets are merged into {@code TableDesc#getJobSecrets()}
+   * for {@link org.apache.hadoop.hive.ql.plan.PlanUtils#configureJobConf} (HIVE-20651).
+   */
+  static void refreshVendedCredentialsIfMissing(TableDesc tableDesc, JobConf jobConf, Configuration configuration) {
+    if (tableDesc == null || tableDesc.getProperties() == null ||
+        hasSerializedCredentials(tableDesc.getJobSecrets()) ||
+        StringUtils.isNotBlank(jobConf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))) {
+      return;
+    }
+
+    Properties props = tableDesc.getProperties();
+    if (!requestsVendedCredentials(props, configuration)) {
+      return;
+    }
+
+    String catalogName = props.getProperty(InputFormatConfig.CATALOG_NAME);
+    if (catalogName == null) {
+      return;
+    }
+
+    try {
+      Table table = getTableWithVendedCredentials(props, configuration);
+      Map<String, String> jobProps = new LinkedHashMap<>();
+      Map<String, String> secrets = new LinkedHashMap<>();
+      propagateToJob(table, catalogName, jobProps, secrets, configuration);
+      jobProps.forEach(jobConf::set);
+      mergeJobSecrets(tableDesc, secrets);
+    } catch (NoSuchTableException ex) {
+      // Table may not exist yet for CTAS; credentials will not be available.
+    }
+  }
+
+  /** Copies {@code jobSecrets} onto {@code jobConf} so the Tez driver can use S3A before tasks start. */
+  static void applyJobSecretsToJobConf(TableDesc tableDesc, JobConf jobConf) {
+    Map<String, String> secrets = tableDesc.getJobSecrets();
+    if (secrets != null) {
+      secrets.forEach(jobConf::set);
+    }
+  }
+
+  private static boolean hasSerializedCredentials(Map<String, String> jobSecrets) {
+    return jobSecrets != null &&
+        StringUtils.isNotBlank(jobSecrets.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS));
+  }
+
+  private static void mergeJobSecrets(TableDesc tableDesc, Map<String, String> secrets) {
+    if (secrets.isEmpty()) {
+      return;
+    }
+    Map<String, String> existing = tableDesc.getJobSecrets();
+    if (existing == null) {
+      tableDesc.setJobSecrets(new LinkedHashMap<>(secrets));
+    } else {
+      secrets.forEach(existing::putIfAbsent);
+    }
+  }
+
+  private static boolean isSecretKey(String icebergKey) {
+    return SECRET_ICEBERG_KEYS.contains(icebergKey);
+  }
+
+  static List<StorageCredential> extractCredentials(Table table) {
+    if (table == null) {
+      return List.of();
+    }
+    FileIO io = table.io();
+    if (io instanceof SupportsStorageCredentials credentialIo) {
+      List<StorageCredential> credentials = credentialIo.credentials();
+      if (credentials != null && !credentials.isEmpty()) {
+        return credentials;
+      }
+    }
+    return credentialsFromFileIoProperties(table, io);
+  }
+
+  private static List<StorageCredential> credentialsFromFileIoProperties(Table table, FileIO io) {
+    Map<String, String> props = io.properties();
+    if (props == null || StringUtils.isBlank(props.get(ACCESS_KEY_ID)) ||
+        StringUtils.isBlank(props.get(SECRET_ACCESS_KEY))) {
+      return List.of();
+    }
+    Map<String, String> config = new LinkedHashMap<>();
+    putIfPresent(config, props, ACCESS_KEY_ID);
+    putIfPresent(config, props, SECRET_ACCESS_KEY);
+    putIfPresent(config, props, SESSION_TOKEN);
+    putIfPresent(config, props, ENDPOINT);
+    putIfPresent(config, props, PATH_STYLE_ACCESS);
+    putIfPresent(config, props, "client.region");
+    return List.of(StorageCredential.create(credentialPrefix(table), config));
+  }
+
+  private static void putIfPresent(Map<String, String> target, Map<String, String> source, String key) {
+    if (source.containsKey(key) && StringUtils.isNotBlank(source.get(key))) {
+      target.put(key, source.get(key));
+    }
+  }
+
+  private static String credentialPrefix(Table table) {
+    String location = table.location();
+    if (StringUtils.isBlank(location)) {
+      return "";
+    }
+    String bucket = bucketFromPrefix(location);
+    if (bucket != null) {
+      return "s3://" + bucket + "/";
+    }
+    return location.endsWith("/") ? location : location + "/";
+  }
+
+  /**
+   * REST catalogs vend credentials together with S3 connectivity settings such as endpoint
+   * and path-style access. These settings reflect the catalog's network view and may reference
+   * hosts that are not reachable from Hive (for example, an internal {@code minio:9000}
+   * hostname).
+   *
+   * Catalog S3 properties configured in the Hive session (for example,
+   * {@code iceberg.catalog.ice01.s3.endpoint}) override the corresponding vended connectivity
+   * settings so the driver and executors use reachable endpoints. Vended credentials are
+   * preserved; only non-secret connectivity properties are overridden.
+   */
+  private static List<StorageCredential> withConfigurationOverrides(
+      String catalogName, List<StorageCredential> credentials, Configuration conf) {
+
+    if (credentials.isEmpty() || conf == null || catalogName == null) {
+      return credentials;
+    }
+
+    List<StorageCredential> updated = Lists.newArrayListWithCapacity(credentials.size());
+    for (StorageCredential credential : credentials) {
+      Map<String, String> credsConfig = new LinkedHashMap<>(credential.config());
+      applyCatalogConfigOverrides(catalogName, credsConfig, conf);
+      updated.add(StorageCredential.create(credential.prefix(), credsConfig));
+    }
+
+    return updated;
+  }
+
+  /**
+   * Applies session-level catalog overrides for {@link #ENDPOINT} and
+   * {@link #PATH_STYLE_ACCESS} to the given credential configuration.
+   *
+   * <p>Override values are read from {@code conf} using
+   * {@link IcebergCatalogProperties#catalogPropertyConfigKey(String, String)}.
+   * If a non-blank override is configured, it replaces the corresponding value
+   * in {@code config}; otherwise the existing (vended) value is retained.
+   */
+  private static void applyCatalogConfigOverrides(
+      String catalogName, Map<String, String> config, Configuration conf) {
+    for (String icebergKey : List.of(ENDPOINT, PATH_STYLE_ACCESS)) {
+      String override =
+          conf.get(IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey));
+      if (StringUtils.isNotBlank(override)) {
+        config.put(icebergKey, override);
+      }
+    }
+  }
+
+  private static String resolveCredentialValue(
+      String catalogName, String icebergKey, String vendedValue, Configuration conf) {
+    if (conf == null || catalogName == null) {
+      return vendedValue;
+    }
+    String override =
+        conf.get(IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey));
+    return StringUtils.isNotBlank(override) ? override : vendedValue;
+  }
+
+  @SuppressWarnings("java:S1075") // S3 URI path separator, not a filesystem path
+  private static String bucketFromPrefix(String prefix) {
+    if (StringUtils.isBlank(prefix)) {
+      return null;
+    }
+    try {
+      URI uri = new URI(prefix.endsWith("/") ? prefix : prefix + "/");
+      if ("s3".equalsIgnoreCase(uri.getScheme()) || "s3a".equalsIgnoreCase(uri.getScheme())) {
+        return uri.getHost();
+      }
+    } catch (Exception ignored) {
+      // fall through
+    }
+    if (prefix.startsWith("s3://") || prefix.startsWith("s3a://")) {
+      String withoutScheme = prefix.substring(prefix.indexOf("://") + 3);
+      int slash = withoutScheme.indexOf('/');
+      return slash >= 0 ? withoutScheme.substring(0, slash) : withoutScheme;
+    }
+    return null;
+  }
+
+  private static String toS3aBucketProperty(String bucket, String icebergKey) {
+    String bucketPrefix = "fs.s3a.bucket." + bucket + ".";
+    return switch (icebergKey) {
+      case ACCESS_KEY_ID -> bucketPrefix + "access.key";
+      case SECRET_ACCESS_KEY -> bucketPrefix + "secret.key";
+      case SESSION_TOKEN -> bucketPrefix + "session.token";
+      case ENDPOINT -> bucketPrefix + "endpoint";
+      case PATH_STYLE_ACCESS -> bucketPrefix + "path.style.access";
+      case "client.region" -> bucketPrefix + "endpoint.region";

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3577136057


##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   i think test would fail if we remove the config. We need to propagate secrets in the FetchOperator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4424174326

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4493702799

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4626445117

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [11 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559693131


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:
##########
@@ -861,36 +861,44 @@ private static void configureJobPropertiesForStorageHandler(boolean input,
       if (storageHandler != null) {
         Map<String, String> jobProperties = new LinkedHashMap<>();
         Map<String, String> jobSecrets = new LinkedHashMap<>();
-        if(input) {
-            try {
-                storageHandler.configureInputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureInputJobProperties not found "+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
-
-            try{
-              storageHandler.configureInputJobCredentials(
-                tableDesc,
-                jobSecrets);
-            } catch(AbstractMethodError e) {
-              // ignore
-              LOG.info("configureInputJobSecrets not found");
-            }
+        if (input) {
+          try {
+            storageHandler.configureInputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureInputJobProperties not found "+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(
+              tableDesc,
+              jobSecrets);
+          } catch (AbstractMethodError e) {
+            // ignore
+            LOG.info("configureInputJobSecrets not found");
+          }
         }
         else {
-            try {
-                storageHandler.configureOutputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureOutputJobProperties not found"+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
+          try {
+            storageHandler.configureOutputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureOutputJobProperties not found"+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(

Review Comment:
   maybe configureOutputJobCredentials ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3573998349


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   Fixed by your 'Strip → pass → restore' patch.



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1681,7 +1705,11 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
     PartitionSpec spec;
     String bytes;
     try {
-      Table table = IcebergTableUtil.getTable(configuration, props);
+      boolean isVendedCredentials =
+          IcebergVendedCredentialUtil.requestsVendedCredentials(props, configuration);
+      Table table = isVendedCredentials ?

Review Comment:
   Same, fixed by your 'Strip → pass → restore' patch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3577296875


##########
iceberg/iceberg-handler/src/test/queries/positive/iceberg_rest_catalog_gravitino.q:
##########
@@ -24,9 +28,13 @@ set hive.stats.autogather=false;
 set metastore.client.impl=org.apache.iceberg.hive.client.HiveRESTCatalogClient;
 set metastore.catalog.default=ice01;
 set iceberg.catalog.ice01.type=rest;
+set iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation=vended-credentials;
+--! Vended credentials travel via DAG credentials (HIVE-20651); an in-process fetch task cannot
+--! restore them, so fetch conversion must be off for credential-vending catalogs.
+set hive.fetch.task.conversion=none;

Review Comment:
   ````
   Utilities.copyTableJobPropertiesToConf(currDesc.getTableDesc(), job);
   -------------------
   if (currDesc.getTableDesc().getJobSecrets() != null) {
     currDesc.getTableDesc().getJobSecrets().forEach(job::set);
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3580737768


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,499 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Base64;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConfUtil;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.BaseMetadataTable;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.aws.AwsClientProperties;
+import org.apache.iceberg.aws.s3.S3FileIOProperties;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ *
+ * <p>Limitation: credentials are minted at compile/job launch and are not refreshed — work that
+ * outlives the vended token lifetime fails authentication.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table; its catalog-qualified name keys the credentials blob,
+   *        since REST catalogs mint credentials per table (the table pointer carries the same
+   *        name, so executors resolve the matching key)
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.vendedCredentialsKey(table.name()),
+          serializeToSingleLineBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey, conf)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey, conf)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob.
+   * First table wins on a shared bucket, matching the non-secret entries, so an endpoint and its
+   * key always come from the same credential. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,

Review Comment:
   seems format specific wonder if we should refactor? how do we even know what type of storage vended credentials are for? 
   is it possible to have some generic abstraction?
   https://gravitino.apache.org/docs/1.3.0/security/credential-vending



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4569799685

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559599551


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1681,7 +1705,11 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
     PartitionSpec spec;
     String bytes;
     try {
-      Table table = IcebergTableUtil.getTable(configuration, props);
+      boolean isVendedCredentials =
+          IcebergVendedCredentialUtil.requestsVendedCredentials(props, configuration);
+      Table table = isVendedCredentials ?
+          IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, configuration) :

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls `HiveTableUtil.serializeTable(table, ...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3560544693


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";

Review Comment:
   why not reuse from S3FileIOProperties?
   ````
    private static final ImmutableSet<String> SECRET_ICEBERG_KEYS = ImmutableSet.of(
         S3FileIOProperties.ACCESS_KEY_ID, S3FileIOProperties.SECRET_ACCESS_KEY, S3FileIOProperties.SESSION_TOKEN);
   
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4557612693

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [30 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4964049065

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3428759632


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   Most Iceberg clients don't need to ser/de credentials on their own, probably. We need it. That's because we serialize an Iceberg table in Hadoop's configuration as an intermediate expression? I guess we should store the credentials as they are and just restore them, without refining or normalizing the content on the Hive side



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4974303433

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559310656


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -335,32 +350,10 @@ public void commitJob(JobContext originalContext) {
   @Override
   public void configureJobConf(TableDesc tableDesc, JobConf jobConf) {
     setCommonJobConf(jobConf);
-    if (tableDesc != null && tableDesc.getProperties() != null &&
-        tableDesc.getProperties().get(InputFormatConfig.OPERATION_TYPE_PREFIX + tableDesc.getTableName()) != null) {
-      String tableName = tableDesc.getTableName();
-      String opKey = InputFormatConfig.OPERATION_TYPE_PREFIX + tableName;
-      // set operation type into job conf too
-      jobConf.set(opKey, tableDesc.getProperties().getProperty(opKey));
-      Preconditions.checkArgument(!tableName.contains(TABLE_NAME_SEPARATOR),
-          "Can not handle table " + tableName + ". Its name contains '" + TABLE_NAME_SEPARATOR + "'");
-      if (HiveCustomStorageHandlerUtils.getWriteOperation(tableDesc.getProperties()::getProperty, tableName) != null) {
-        HiveCustomStorageHandlerUtils.setWriteOperation(jobConf, tableName,
-            Operation.valueOf(tableDesc.getProperties().getProperty(
-                HiveCustomStorageHandlerUtils.WRITE_OPERATION_CONFIG_PREFIX + tableName)));
-      }
-      boolean isMergeTaskEnabled = Boolean.parseBoolean(tableDesc.getProperty(
-          HiveCustomStorageHandlerUtils.MERGE_TASK_ENABLED + tableName));
-      if (isMergeTaskEnabled) {
-        HiveCustomStorageHandlerUtils.setMergeTaskEnabled(jobConf, tableName, true);
-      }
-      String tables = jobConf.get(InputFormatConfig.OUTPUT_TABLES);
-      tables = (tables == null) ? tableName : tables + TABLE_NAME_SEPARATOR + tableName;
-      jobConf.set(InputFormatConfig.OUTPUT_TABLES, tables);
-
-      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);
-      if (catalogName != null) {
-        jobConf.set(InputFormatConfig.TABLE_CATALOG_PREFIX + tableName, catalogName);
-      }
+    configureOutputTableJobConf(tableDesc, jobConf);
+    if (IcebergVendedCredentialUtil.requestsVendedCredentials(tableDesc.getProperties(), conf)) {
+      IcebergVendedCredentialUtil.refreshVendedCredentialsIfMissing(tableDesc, jobConf, conf);
+      IcebergVendedCredentialUtil.applyJobSecretsToJobConf(tableDesc, jobConf);

Review Comment:
   doesn't this copies the whole secrets map — raw fs.s3a.bucket.<b>.access.key/secret.key/session.token plus the base64 credential blob — into the plain JobCon ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559636516


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -284,6 +284,21 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
     return null;
   }
 
+  @Override
+  public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {

Review Comment:
   what about `configureOutputJobCredentials` (for output tables) ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there including things like HiveRESTCatalogClient, VendedCredentials, client creation, etc ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559599551


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1681,7 +1705,11 @@ static void overlayTableProperties(Configuration configuration, TableDesc tableD
     PartitionSpec spec;
     String bytes;
     try {
-      Table table = IcebergTableUtil.getTable(configuration, props);
+      boolean isVendedCredentials =
+          IcebergVendedCredentialUtil.requestsVendedCredentials(props, configuration);
+      Table table = isVendedCredentials ?
+          IcebergVendedCredentialUtil.getTableWithVendedCredentials(props, configuration) :

Review Comment:
   should we strip vended secrets from serialized table blob ? 
   overlayTableProperties now loads the table with vended credentials and then calls `HiveTableUtil.serializeTable(table, ...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559636516


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -284,6 +284,21 @@ public HiveAuthorizationProvider getAuthorizationProvider() {
     return null;
   }
 
+  @Override
+  public void configureInputJobCredentials(TableDesc tableDesc, Map<String, String> secrets) {

Review Comment:
   what about `configureOutputJobCredentials` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct — vended credentials in `table.io().properties()` survive ser/de to executors. We still zneed to extract those credentials and stage them in Hive job configuration for two reasons:
   
   1. **Hadoop S3A.** Iceberg FileIO and Hadoop S3A do not share state; S3A only reads Configuration, and Tez/LLAP use S3A for `s3://` paths. At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. **Iceberg FileIO endpoint override.** `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed when REST catalogs vend S3 connection settings for their own network while Hive runs elsewhere. In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint`=`http://minio:9000`, which tasks on the host cannot reach; session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes vended credentials (from job conf or `FileIO.properties()`), replaces only endpoint and path-style with session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still had `http://minio:9000`, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3428674517


##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestIcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,431 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SerializationUtil;
+import org.junit.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests for {@link IcebergVendedCredentialUtil}: copying REST-vended storage credentials into Hive
+ * job configuration at plan time and re-applying them on executors.
+ */
+public class TestIcebergVendedCredentialUtil {
+
+  /**
+   * Tests {@link IcebergVendedCredentialUtil#requestsVendedCredentials(Properties, Configuration)}.
+   *
+   * <p><b>Setup:</b> table properties name catalog {@code ice01}; session {@code conf} has no
+   * delegation header, then gains {@code vended-credentials}.
+   *
+   * <p><b>Expects:</b> {@code false} without the header, {@code true} with it — propagation and
+   * apply logic must not run for ordinary REST catalogs.
+   *
+   * <p><b>Why:</b> Iceberg REST only vends storage credentials when the client sends
+   * {@code X-Iceberg-Access-Delegation: vended-credentials}; Hive mirrors that opt-in via catalog
+   * config so static catalog keys are not replaced unnecessarily.
+   */
+  @Test
+  public void requestsVendedCredentialsRequiresDelegationHeader() {
+    Configuration conf = new Configuration();
+    Properties props = new java.util.Properties();
+    props.setProperty(InputFormatConfig.CATALOG_NAME, "ice01");
+
+    assertThat(IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf)).isFalse();
+
+    conf.set(
+        "iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation",
+        "vended-credentials");
+    assertThat(IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf)).isTrue();
+  }
+
+  /**
+   * Tests {@link IcebergVendedCredentialUtil#propagateToJob} when both {@code jobProperties} and
+   * {@code jobSecrets} are non-null (full compile-time path after both storage-handler hooks).
+   *
+   * <p><b>Setup:</b> vended credential with internal endpoint {@code http://minio:9000}; session
+   * {@code conf} sets {@code iceberg.catalog.ice01.s3.endpoint} to {@code http://host:9000}.
+   *
+   * <p><b>Expects:</b> endpoint and path-style in {@code jobProps} (Iceberg catalog keys and S3A
+   * per-bucket keys) with host endpoint; access/secret and serialized blob in {@code jobSecrets}
+   * only; blob endpoint also host; no secret material in {@code jobProps}.
+   *
+   * <p><b>Why:</b> HIVE-20651 keeps secrets in {@code TableDesc#jobSecrets} (later Hadoop
+   * {@code Credentials}) and non-secrets in job conf. Catalogs often vend an internal S3 endpoint;
+   * HS2 overrides with a reachable host endpoint at propagation time so Tez tasks and Iceberg FileIO
+   * agree on connectivity while still using vended keys.
+   */
+  @Test
+  public void propagateToJobMapsIcebergAndS3aProperties() {
+    Configuration conf = new Configuration();
+    conf.set("iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ENDPOINT, "http://host:9000");
+
+    CredentialFileIO fileIO = new CredentialFileIO();
+    fileIO.setCredentials(
+        List.of(
+            StorageCredential.create(
+                "s3://my-bucket/",
+                Map.of(
+                    IcebergVendedCredentialUtil.ACCESS_KEY_ID, "access",
+                    IcebergVendedCredentialUtil.SECRET_ACCESS_KEY, "secret",
+                    IcebergVendedCredentialUtil.ENDPOINT, "http://minio:9000",
+                    IcebergVendedCredentialUtil.PATH_STYLE_ACCESS, "true"))));
+
+    Table table =
+        new BaseTable(new StaticTableOperations("s3://my-bucket/t", fileIO), "db.t");
+    Map<String, String> jobProps = Maps.newHashMap();
+    Map<String, String> jobSecrets = Maps.newHashMap();
+
+    IcebergVendedCredentialUtil.propagateToJob(table, "ice01", jobProps, jobSecrets, conf);
+
+    assertThat(jobProps)
+        .containsEntry(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ENDPOINT,
+            "http://host:9000")
+        .containsEntry(
+            "fs.s3a.bucket.my-bucket.endpoint",
+            "http://host:9000")
+        .containsEntry(
+            "fs.s3a.bucket.my-bucket.path.style.access",
+            "true")
+        .doesNotContainKey(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)
+        .doesNotContainKey("fs.s3a.bucket.my-bucket.access.key")
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ACCESS_KEY_ID);
+
+    assertThat(jobSecrets)
+        .containsEntry("fs.s3a.bucket.my-bucket.access.key", "access")
+        .containsEntry("fs.s3a.bucket.my-bucket.secret.key", "secret")
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ACCESS_KEY_ID)
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.SECRET_ACCESS_KEY)
+        .satisfies(map ->
+            assertThat(map.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))
+                .isNotBlank());

Review Comment:
   I am feeling the prefix information is gone.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (SerializableTable) into JobConf for Tez/LLAP, and vended credentials on FileIO typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST `loadTable`, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + `S3A` bucket keys) and restore them in `deserializeTable` via `applyFromJobConf`.
   
   On "store as-is / restore as-is" vs the helper methods:
   
   **`resolveCredentialsForApply()`**: At apply time credentials can live in different places depending on context:
   
   - Tez/LLAP tasks: deserialized Table has an empty `FileIO.credentials()`; the source of truth is the serialized blob in task conf (see `applyFromJobConfRestoresCredentialsOnExecutor`).
   - HS2 commit: the table often still has credentials from the earlier loadTable in QueryState; we prefer those over re-reading the blob.
   - Fallback: `extractCredentials(table)` from `FileIO` if both are empty.
   
   **`credentialsFromFileIoProperties()`**: needed at plan/propagate time. Some REST/FileIO implementations populate `s3.access-key-id` / `s3.secret-access-key` in `FileIO.properties()` while credentials() is still empty after loadTable. extractCredentials uses this fallback so propagateToJob doesn't skip vending even though the table load succeeded (extractCredentialsFromFileIoPropertiesWhenCredentialListEmpty). It also backs the last-resort branch in resolveCredentialsForApply.
   
   **`withConfigurationOverrides`**: this is the part that does mutate vended credentials. REST catalogs often vend connectivity settings from their network view (e.g. `http://minio:9000` inside Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint`=`http://host:9000`). We override only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on FileIO from `loadTable`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3487369152


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   > vended credentials on FileIO typically don't survive that round-trip
   
   Is this tested? I presume it is part of serializable objects



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3455665616


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   You're right that most Iceberg clients don't need to ser/de credentials themselves. Hive does, because we serialize the Iceberg Table (`SerializableTable`) into `JobConf` for Tez/LLAP, and vended credentials on `FileIO` typically don't survive that round-trip. Executors rebuild the table from job conf and don't re-run REST loadTable, so we propagate credentials separately (`VENDED_STORAGE_CREDENTIALS` + S3A bucket keys) and restore them in `deserializeTable` via `applyFromJobConf`.
   
   The main place we mutate vended credential content is `withConfigurationOverrides()` method. REST catalogs can vend connectivity settings from their network view (e.g. `http://minio:9000` when the catalog runs in Docker), while Hive session config sets a host-reachable endpoint (`iceberg.catalog.ice01.s3.endpoint=http://host:9000`). That method overrides only non-secret fields (`s3.endpoint`, `s3.path-style-access`) so Iceberg `FileIO` and S3A agree on connectivity; vended keys are preserved. It runs at both store time (`propagateToJob`, so the blob on executors is self-contained) and restore time (`applyFromJobConf`, e.g. when commit still has the catalog-internal endpoint on `FileIO` from loadTable).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3581426435


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,499 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Base64;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConfUtil;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.BaseMetadataTable;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.aws.AwsClientProperties;
+import org.apache.iceberg.aws.s3.S3FileIOProperties;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ *
+ * <p>Limitation: credentials are minted at compile/job launch and are not refreshed — work that
+ * outlives the vended token lifetime fails authentication.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table; its catalog-qualified name keys the credentials blob,
+   *        since REST catalogs mint credentials per table (the table pointer carries the same
+   *        name, so executors resolve the matching key)
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.vendedCredentialsKey(table.name()),
+          serializeToSingleLineBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey, conf)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey, conf)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob.
+   * First table wins on a shared bucket, matching the non-secret entries, so an endpoint and its
+   * key always come from the same credential. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,

Review Comment:
   Agreed. Iceberg credential transport is already generic, we serialize/restore the `StorageCredential` list for Iceberg `FileIO` on executors. The S3-specific part is the Hadoop propagation (`toS3aBucketProperty()` / `addSecretCredentialEntry()`), which maps vended config to `fs.s3a.bucket.*` because Tez/LLAP still use Hadoop FileSystem for some `s3://` paths.
   
   This PR intentionally keeps that scope to S3 (Gravitino q-test + S3FileIO). Can we handle provider-aware propagation for GCS/ADLS/OSS in a follow-up, deriving storage type from `StorageCredential.prefix()` and mapping to the right Hadoop FS / Iceberg IO settings per provider?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct - vended credentials in `table.io().properties()` survive ser/de to executors. But I believe we still need parsing these credentials and moving them to Hive configuration for the following 2 reasons:
   
   1. We still need to put credentials in JobConf because Iceberg FileIO and Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use S3A for many s3:// paths.
   
   2. `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed because there can be setups where REST catalogs vend S3 connection settings for their own environment while Hive is running in a different network. For example, In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach. Hive session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes the vended credentials, replaces only the endpoint and path-style fields with the session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still contained `http://minio:9000`, and need to be overridden, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there including rest of things like VendedCredentials, client creation, etc ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3558992551


##########
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/RestAccessDelegationMode.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hive;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Values for the Iceberg REST catalog {@code X-Iceberg-Access-Delegation} request header. The header
+ * accepts a comma-separated list of these modes; configure via
+ * {@link IcebergCatalogProperties#REST_ACCESS_DELEGATION_HEADER_PROPERTY}.
+ *
+ * @see <a href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml">REST catalog spec</a>
+ */
+public enum RestAccessDelegationMode {

Review Comment:
   should we introduce new module `rest-catalog-client` and move it there including rest of things like VendedCredentials, client creation, etc ?
   
   or move it in handler, i don't think it belongs to hms-catalog



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3559693131


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:
##########
@@ -861,36 +861,44 @@ private static void configureJobPropertiesForStorageHandler(boolean input,
       if (storageHandler != null) {
         Map<String, String> jobProperties = new LinkedHashMap<>();
         Map<String, String> jobSecrets = new LinkedHashMap<>();
-        if(input) {
-            try {
-                storageHandler.configureInputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureInputJobProperties not found "+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
-
-            try{
-              storageHandler.configureInputJobCredentials(
-                tableDesc,
-                jobSecrets);
-            } catch(AbstractMethodError e) {
-              // ignore
-              LOG.info("configureInputJobSecrets not found");
-            }
+        if (input) {
+          try {
+            storageHandler.configureInputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureInputJobProperties not found "+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(
+              tableDesc,
+              jobSecrets);
+          } catch (AbstractMethodError e) {
+            // ignore
+            LOG.info("configureInputJobSecrets not found");
+          }
         }
         else {
-            try {
-                storageHandler.configureOutputJobProperties(
-                  tableDesc,
-                  jobProperties);
-            } catch(AbstractMethodError e) {
-                LOG.info("configureOutputJobProperties not found"+
-                    "using configureTableJobProperties",e);
-                storageHandler.configureTableJobProperties(tableDesc, jobProperties);
-            }
+          try {
+            storageHandler.configureOutputJobProperties(
+              tableDesc,
+              jobProperties);
+          } catch (AbstractMethodError e) {
+            LOG.info("configureOutputJobProperties not found"+
+                "using configureTableJobProperties",e);
+            storageHandler.configureTableJobProperties(tableDesc, jobProperties);
+          }
+
+          try {
+            storageHandler.configureInputJobCredentials(

Review Comment:
   is that really needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3560551796


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.net.URI;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hive.IcebergCatalogProperties;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.SerializationUtil;
+import org.springframework.util.CollectionUtils;
+
+/**
+ * Propagates vended storage credentials from an Iceberg {@link Table}'s {@link FileIO} to Hive job
+ * configuration so Tez/LLAP executors can access object storage without static catalog keys.
+ */
+public final class IcebergVendedCredentialUtil {
+
+  public static final String ACCESS_KEY_ID = "s3.access-key-id";
+  public static final String SECRET_ACCESS_KEY = "s3.secret-access-key";
+  public static final String SESSION_TOKEN = "s3.session-token";
+  public static final String ENDPOINT = "s3.endpoint";
+  public static final String PATH_STYLE_ACCESS = "s3.path-style-access";
+
+  private static final ImmutableSet<String> SECRET_ICEBERG_KEYS = ImmutableSet.of(
+      ACCESS_KEY_ID, SECRET_ACCESS_KEY, SESSION_TOKEN);
+
+  private IcebergVendedCredentialUtil() {
+  }
+
+  /**
+   * Copies vended credentials from the table FileIO into Hive job configuration.
+   *
+   * <p>Follows the HIVE-20651 split used by {@code JdbcStorageHandler}: sensitive values (including
+   * the serialized {@link StorageCredential} list) go to {@code jobSecrets}; non-secret config such
+   * as endpoint and path-style access go to {@code jobProperties}.
+   *
+   * @param table loaded Iceberg table
+   * @param catalogName Hive catalog name ({@link InputFormatConfig#CATALOG_NAME})
+   * @param jobProperties Tez/MR non-secret job properties; may be {@code null}
+   * @param jobSecrets sensitive keys and serialized credentials; may be {@code null}
+   * @param conf session conf used to preserve host-side endpoint overrides
+   */
+  public static void propagateToJob(Table table, String catalogName, Map<String, String> jobProperties,
+      Map<String, String> jobSecrets, Configuration conf) {
+
+    List<StorageCredential> credentials =
+        withConfigurationOverrides(catalogName, extractCredentials(table), conf);
+
+    if (credentials.isEmpty()) {
+      return;
+    }
+
+    if (jobSecrets != null) {
+      jobSecrets.put(
+          InputFormatConfig.VENDED_STORAGE_CREDENTIALS,
+          SerializationUtil.serializeToBase64(Lists.newArrayList(credentials)));
+    }
+
+    for (StorageCredential credential : credentials) {
+      addCredentialEntries(catalogName, credential, jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Writes each key in one vended {@link StorageCredential} into job configuration.
+   *
+   * <p>Derives the S3 bucket from {@link StorageCredential#prefix()} and delegates to
+   * {@link #addCredentialEntry} for every entry in {@link StorageCredential#config()}, which maps
+   * Iceberg keys to catalog-level and per-bucket S3A job properties or secrets.
+   */
+  private static void addCredentialEntries(String catalogName, StorageCredential credential,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    String bucket = bucketFromPrefix(credential.prefix());
+    for (Map.Entry<String, String> entry : credential.config().entrySet()) {
+      addCredentialEntry(
+          catalogName, bucket, entry.getKey(), entry.getValue(), jobProperties, jobSecrets, conf);
+    }
+  }
+
+  /**
+   * Routes one Iceberg credential config entry into job properties or secrets.
+   *
+   * <p>Skips blank values, applies session catalog overrides via {@link #resolveCredentialValue},
+   * then sends non-secret keys (endpoint, path-style access, etc.) to {@code jobProperties} and
+   * secret keys (access key, secret key, session token) to {@code jobSecrets}. Either map may be
+   * {@code null} when {@link #propagateToJob} is called for only properties or only secrets.
+   */
+  private static void addCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties, Map<String, String> jobSecrets, Configuration conf) {
+
+    if (StringUtils.isBlank(value)) {
+      return;
+    }
+    String resolvedValue = resolveCredentialValue(catalogName, icebergKey, value, conf);
+
+    if (jobProperties != null && !isSecretKey(icebergKey)) {
+      addNonSecretCredentialEntry(catalogName, bucket, icebergKey, resolvedValue, jobProperties);
+    }
+
+    if (jobSecrets != null && isSecretKey(icebergKey)) {
+      addSecretCredentialEntry(bucket, icebergKey, resolvedValue, jobSecrets);
+    }
+  }
+
+  /**
+   * Adds one non-secret vended value to {@code jobProperties} for Iceberg and Hadoop S3A.
+   *
+   * <p>When {@code catalogName} is set, writes {@code iceberg.catalog.&lt;catalog&gt;.&lt;key&gt;}.
+   * When {@code bucket} is set, also writes the matching {@code fs.s3a.bucket.&lt;bucket&gt;.*}
+   * key if {@link #toS3aBucketProperty} maps the Iceberg key.
+   */
+  private static void addNonSecretCredentialEntry(String catalogName, String bucket, String icebergKey, String value,
+      Map<String, String> jobProperties) {
+
+    if (catalogName != null) {
+      String catalogConfigKey =
+          IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey);
+      jobProperties.putIfAbsent(catalogConfigKey, value);
+    }
+
+    if (bucket != null) {
+      String s3aKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aKey != null) {
+        jobProperties.putIfAbsent(s3aKey, value);
+      }
+    }
+  }
+
+  /** Writes Hadoop S3A per-bucket keys only; Iceberg secrets are carried in the serialized blob. */
+  private static void addSecretCredentialEntry(String bucket, String icebergKey, String value,
+      Map<String, String> jobSecrets) {
+    if (bucket != null) {
+      String s3aSecretKey = toS3aBucketProperty(bucket, icebergKey);
+      if (s3aSecretKey != null) {
+        jobSecrets.put(s3aSecretKey, value);
+      }
+    }
+  }
+
+  /**
+   * Applies vended credentials to the table FileIO, merging session/catalog conf overrides (e.g. S3 endpoint).
+   * Used on executors after deserialization and on HS2 commit when the table is taken from query state.
+   */
+  public static void applyFromJobConf(Table table, Configuration conf) {
+    if (table == null || conf == null) {
+      return;
+    }
+
+    String catalogName = conf.get(InputFormatConfig.CATALOG_NAME);
+    if (shouldSkipApplyFromJobConf(catalogName, conf)) {
+      return;
+    }
+
+    FileIO io = table.io();
+    if (!(io instanceof SupportsStorageCredentials credentialIo)) {
+      return;
+    }
+
+    List<StorageCredential> credentials = resolveCredentialsForApply(table, credentialIo, conf);
+    if (credentials != null && !credentials.isEmpty()) {
+      credentialIo.setCredentials(withConfigurationOverrides(catalogName, credentials, conf));
+    }
+  }
+
+  /**
+   * Returns true when the job carries no serialized vended credentials and the catalog is not
+   * configured for credential vending (no {@code vended-credentials} REST delegation header).
+   * Otherwise apply may restore credentials from the job conf or from the table FileIO.
+   */
+  private static boolean shouldSkipApplyFromJobConf(String catalogName, Configuration conf) {
+    return StringUtils.isBlank(conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)) &&
+        !IcebergCatalogProperties.requestsVendedCredentials(catalogName, conf);
+  }
+
+  /**
+   * Chooses which vended credentials {@link #applyFromJobConf} should install on the FileIO.
+   *
+   * <p>Uses the first non-empty source: credentials already on the FileIO (typical on HS2 after
+   * table load), else the base64 list from {@link InputFormatConfig#VENDED_STORAGE_CREDENTIALS} on
+   * the task {@code conf} (propagated at plan time), else {@link #extractCredentials(Table)} from
+   * the table (including FileIO property fallbacks).
+   */
+  private static List<StorageCredential> resolveCredentialsForApply(
+      Table table, SupportsStorageCredentials credentialIo, Configuration conf) {
+
+    List<StorageCredential> credentials = credentialIo.credentials();
+    if (CollectionUtils.isEmpty(credentials)) {
+      String serialized = conf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS);
+      if (StringUtils.isNotBlank(serialized)) {
+        @SuppressWarnings("unchecked")
+        List<StorageCredential> deserialized =
+            SerializationUtil.deserializeFromBase64(serialized);
+        credentials = deserialized;
+      }
+    }
+
+    if (credentials == null || credentials.isEmpty()) {
+      credentials = extractCredentials(table);
+    }
+    return credentials;
+  }
+
+  /**
+   * Returns true when the table catalog is configured for REST vended storage credentials.
+   */
+  static boolean requestsVendedCredentials(Properties properties, Configuration configuration) {
+    if (properties == null) {
+      return false;
+    }
+    return IcebergCatalogProperties.requestsVendedCredentials(
+        properties.getProperty(InputFormatConfig.CATALOG_NAME), configuration);
+  }
+
+  /**
+   * Loads a table and, if needed, bypasses the query-level cache so REST vended credentials are present on the FileIO.
+   * When vended credentials are not requested for the catalog, returns the cached table without an extra load.
+   */
+  static Table getTableWithVendedCredentials(Properties properties, Configuration configuration) {
+    Table table = IcebergTableUtil.getTable(configuration, properties);
+    if (requestsVendedCredentials(properties, configuration) && extractCredentials(table).isEmpty()) {
+      table = IcebergTableUtil.getTable(configuration, properties, true);
+    }
+    return table;
+  }
+
+  /**
+   * Reloads vended credentials at job launch when compile-time propagation missed them.
+   * Non-secret config is written to {@code jobConf}; secrets are merged into {@code TableDesc#getJobSecrets()}
+   * for {@link org.apache.hadoop.hive.ql.plan.PlanUtils#configureJobConf} (HIVE-20651).
+   */
+  static void refreshVendedCredentialsIfMissing(TableDesc tableDesc, JobConf jobConf, Configuration configuration) {
+    if (tableDesc == null || tableDesc.getProperties() == null ||
+        hasSerializedCredentials(tableDesc.getJobSecrets()) ||
+        StringUtils.isNotBlank(jobConf.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))) {
+      return;
+    }
+
+    Properties props = tableDesc.getProperties();
+    if (!requestsVendedCredentials(props, configuration)) {
+      return;
+    }
+
+    String catalogName = props.getProperty(InputFormatConfig.CATALOG_NAME);
+    if (catalogName == null) {
+      return;
+    }
+
+    try {
+      Table table = getTableWithVendedCredentials(props, configuration);
+      Map<String, String> jobProps = new LinkedHashMap<>();
+      Map<String, String> secrets = new LinkedHashMap<>();
+      propagateToJob(table, catalogName, jobProps, secrets, configuration);
+      jobProps.forEach(jobConf::set);
+      mergeJobSecrets(tableDesc, secrets);
+    } catch (NoSuchTableException ex) {
+      // Table may not exist yet for CTAS; credentials will not be available.
+    }
+  }
+
+  /** Copies {@code jobSecrets} onto {@code jobConf} so the Tez driver can use S3A before tasks start. */
+  static void applyJobSecretsToJobConf(TableDesc tableDesc, JobConf jobConf) {
+    Map<String, String> secrets = tableDesc.getJobSecrets();
+    if (secrets != null) {
+      secrets.forEach(jobConf::set);
+    }
+  }
+
+  private static boolean hasSerializedCredentials(Map<String, String> jobSecrets) {
+    return jobSecrets != null &&
+        StringUtils.isNotBlank(jobSecrets.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS));
+  }
+
+  private static void mergeJobSecrets(TableDesc tableDesc, Map<String, String> secrets) {
+    if (secrets.isEmpty()) {
+      return;
+    }
+    Map<String, String> existing = tableDesc.getJobSecrets();
+    if (existing == null) {
+      tableDesc.setJobSecrets(new LinkedHashMap<>(secrets));
+    } else {
+      secrets.forEach(existing::putIfAbsent);
+    }
+  }
+
+  private static boolean isSecretKey(String icebergKey) {
+    return SECRET_ICEBERG_KEYS.contains(icebergKey);
+  }
+
+  static List<StorageCredential> extractCredentials(Table table) {
+    if (table == null) {
+      return List.of();
+    }
+    FileIO io = table.io();
+    if (io instanceof SupportsStorageCredentials credentialIo) {
+      List<StorageCredential> credentials = credentialIo.credentials();
+      if (credentials != null && !credentials.isEmpty()) {
+        return credentials;
+      }
+    }
+    return credentialsFromFileIoProperties(table, io);
+  }
+
+  private static List<StorageCredential> credentialsFromFileIoProperties(Table table, FileIO io) {
+    Map<String, String> props = io.properties();
+    if (props == null || StringUtils.isBlank(props.get(ACCESS_KEY_ID)) ||
+        StringUtils.isBlank(props.get(SECRET_ACCESS_KEY))) {
+      return List.of();
+    }
+    Map<String, String> config = new LinkedHashMap<>();
+    putIfPresent(config, props, ACCESS_KEY_ID);
+    putIfPresent(config, props, SECRET_ACCESS_KEY);
+    putIfPresent(config, props, SESSION_TOKEN);
+    putIfPresent(config, props, ENDPOINT);
+    putIfPresent(config, props, PATH_STYLE_ACCESS);
+    putIfPresent(config, props, "client.region");
+    return List.of(StorageCredential.create(credentialPrefix(table), config));
+  }
+
+  private static void putIfPresent(Map<String, String> target, Map<String, String> source, String key) {
+    if (source.containsKey(key) && StringUtils.isNotBlank(source.get(key))) {
+      target.put(key, source.get(key));
+    }
+  }
+
+  private static String credentialPrefix(Table table) {
+    String location = table.location();
+    if (StringUtils.isBlank(location)) {
+      return "";
+    }
+    String bucket = bucketFromPrefix(location);
+    if (bucket != null) {
+      return "s3://" + bucket + "/";
+    }
+    return location.endsWith("/") ? location : location + "/";
+  }
+
+  /**
+   * REST catalogs vend credentials together with S3 connectivity settings such as endpoint
+   * and path-style access. These settings reflect the catalog's network view and may reference
+   * hosts that are not reachable from Hive (for example, an internal {@code minio:9000}
+   * hostname).
+   *
+   * Catalog S3 properties configured in the Hive session (for example,
+   * {@code iceberg.catalog.ice01.s3.endpoint}) override the corresponding vended connectivity
+   * settings so the driver and executors use reachable endpoints. Vended credentials are
+   * preserved; only non-secret connectivity properties are overridden.
+   */
+  private static List<StorageCredential> withConfigurationOverrides(
+      String catalogName, List<StorageCredential> credentials, Configuration conf) {
+
+    if (credentials.isEmpty() || conf == null || catalogName == null) {
+      return credentials;
+    }
+
+    List<StorageCredential> updated = Lists.newArrayListWithCapacity(credentials.size());
+    for (StorageCredential credential : credentials) {
+      Map<String, String> credsConfig = new LinkedHashMap<>(credential.config());
+      applyCatalogConfigOverrides(catalogName, credsConfig, conf);
+      updated.add(StorageCredential.create(credential.prefix(), credsConfig));
+    }
+
+    return updated;
+  }
+
+  /**
+   * Applies session-level catalog overrides for {@link #ENDPOINT} and
+   * {@link #PATH_STYLE_ACCESS} to the given credential configuration.
+   *
+   * <p>Override values are read from {@code conf} using
+   * {@link IcebergCatalogProperties#catalogPropertyConfigKey(String, String)}.
+   * If a non-blank override is configured, it replaces the corresponding value
+   * in {@code config}; otherwise the existing (vended) value is retained.
+   */
+  private static void applyCatalogConfigOverrides(
+      String catalogName, Map<String, String> config, Configuration conf) {
+    for (String icebergKey : List.of(ENDPOINT, PATH_STYLE_ACCESS)) {
+      String override =
+          conf.get(IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey));
+      if (StringUtils.isNotBlank(override)) {
+        config.put(icebergKey, override);
+      }
+    }
+  }
+
+  private static String resolveCredentialValue(
+      String catalogName, String icebergKey, String vendedValue, Configuration conf) {
+    if (conf == null || catalogName == null) {
+      return vendedValue;
+    }
+    String override =
+        conf.get(IcebergCatalogProperties.catalogPropertyConfigKey(catalogName, icebergKey));
+    return StringUtils.isNotBlank(override) ? override : vendedValue;
+  }
+
+  @SuppressWarnings("java:S1075") // S3 URI path separator, not a filesystem path
+  private static String bucketFromPrefix(String prefix) {
+    if (StringUtils.isBlank(prefix)) {
+      return null;
+    }
+    try {
+      URI uri = new URI(prefix.endsWith("/") ? prefix : prefix + "/");
+      if ("s3".equalsIgnoreCase(uri.getScheme()) || "s3a".equalsIgnoreCase(uri.getScheme())) {
+        return uri.getHost();
+      }
+    } catch (Exception ignored) {
+      // fall through
+    }
+    if (prefix.startsWith("s3://") || prefix.startsWith("s3a://")) {
+      String withoutScheme = prefix.substring(prefix.indexOf("://") + 3);
+      int slash = withoutScheme.indexOf('/');
+      return slash >= 0 ? withoutScheme.substring(0, slash) : withoutScheme;
+    }
+    return null;
+  }
+
+  private static String toS3aBucketProperty(String bucket, String icebergKey) {
+    String bucketPrefix = "fs.s3a.bucket." + bucket + ".";
+    return switch (icebergKey) {
+      case ACCESS_KEY_ID -> bucketPrefix + "access.key";
+      case SECRET_ACCESS_KEY -> bucketPrefix + "secret.key";
+      case SESSION_TOKEN -> bucketPrefix + "session.token";
+      case ENDPOINT -> bucketPrefix + "endpoint";
+      case PATH_STYLE_ACCESS -> bucketPrefix + "path.style.access";
+      case "client.region" -> bucketPrefix + "endpoint.region";

Review Comment:
   AwsClientProperties.CLIENT_REGION



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3499799546


##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestIcebergVendedCredentialUtil.java:
##########
@@ -0,0 +1,431 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StaticTableOperations;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.StorageCredential;
+import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.SerializationUtil;
+import org.junit.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests for {@link IcebergVendedCredentialUtil}: copying REST-vended storage credentials into Hive
+ * job configuration at plan time and re-applying them on executors.
+ */
+public class TestIcebergVendedCredentialUtil {
+
+  /**
+   * Tests {@link IcebergVendedCredentialUtil#requestsVendedCredentials(Properties, Configuration)}.
+   *
+   * <p><b>Setup:</b> table properties name catalog {@code ice01}; session {@code conf} has no
+   * delegation header, then gains {@code vended-credentials}.
+   *
+   * <p><b>Expects:</b> {@code false} without the header, {@code true} with it — propagation and
+   * apply logic must not run for ordinary REST catalogs.
+   *
+   * <p><b>Why:</b> Iceberg REST only vends storage credentials when the client sends
+   * {@code X-Iceberg-Access-Delegation: vended-credentials}; Hive mirrors that opt-in via catalog
+   * config so static catalog keys are not replaced unnecessarily.
+   */
+  @Test
+  public void requestsVendedCredentialsRequiresDelegationHeader() {
+    Configuration conf = new Configuration();
+    Properties props = new java.util.Properties();
+    props.setProperty(InputFormatConfig.CATALOG_NAME, "ice01");
+
+    assertThat(IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf)).isFalse();
+
+    conf.set(
+        "iceberg.catalog.ice01.header.X-Iceberg-Access-Delegation",
+        "vended-credentials");
+    assertThat(IcebergVendedCredentialUtil.requestsVendedCredentials(props, conf)).isTrue();
+  }
+
+  /**
+   * Tests {@link IcebergVendedCredentialUtil#propagateToJob} when both {@code jobProperties} and
+   * {@code jobSecrets} are non-null (full compile-time path after both storage-handler hooks).
+   *
+   * <p><b>Setup:</b> vended credential with internal endpoint {@code http://minio:9000}; session
+   * {@code conf} sets {@code iceberg.catalog.ice01.s3.endpoint} to {@code http://host:9000}.
+   *
+   * <p><b>Expects:</b> endpoint and path-style in {@code jobProps} (Iceberg catalog keys and S3A
+   * per-bucket keys) with host endpoint; access/secret and serialized blob in {@code jobSecrets}
+   * only; blob endpoint also host; no secret material in {@code jobProps}.
+   *
+   * <p><b>Why:</b> HIVE-20651 keeps secrets in {@code TableDesc#jobSecrets} (later Hadoop
+   * {@code Credentials}) and non-secrets in job conf. Catalogs often vend an internal S3 endpoint;
+   * HS2 overrides with a reachable host endpoint at propagation time so Tez tasks and Iceberg FileIO
+   * agree on connectivity while still using vended keys.
+   */
+  @Test
+  public void propagateToJobMapsIcebergAndS3aProperties() {
+    Configuration conf = new Configuration();
+    conf.set("iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ENDPOINT, "http://host:9000");
+
+    CredentialFileIO fileIO = new CredentialFileIO();
+    fileIO.setCredentials(
+        List.of(
+            StorageCredential.create(
+                "s3://my-bucket/",
+                Map.of(
+                    IcebergVendedCredentialUtil.ACCESS_KEY_ID, "access",
+                    IcebergVendedCredentialUtil.SECRET_ACCESS_KEY, "secret",
+                    IcebergVendedCredentialUtil.ENDPOINT, "http://minio:9000",
+                    IcebergVendedCredentialUtil.PATH_STYLE_ACCESS, "true"))));
+
+    Table table =
+        new BaseTable(new StaticTableOperations("s3://my-bucket/t", fileIO), "db.t");
+    Map<String, String> jobProps = Maps.newHashMap();
+    Map<String, String> jobSecrets = Maps.newHashMap();
+
+    IcebergVendedCredentialUtil.propagateToJob(table, "ice01", jobProps, jobSecrets, conf);
+
+    assertThat(jobProps)
+        .containsEntry(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ENDPOINT,
+            "http://host:9000")
+        .containsEntry(
+            "fs.s3a.bucket.my-bucket.endpoint",
+            "http://host:9000")
+        .containsEntry(
+            "fs.s3a.bucket.my-bucket.path.style.access",
+            "true")
+        .doesNotContainKey(InputFormatConfig.VENDED_STORAGE_CREDENTIALS)
+        .doesNotContainKey("fs.s3a.bucket.my-bucket.access.key")
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ACCESS_KEY_ID);
+
+    assertThat(jobSecrets)
+        .containsEntry("fs.s3a.bucket.my-bucket.access.key", "access")
+        .containsEntry("fs.s3a.bucket.my-bucket.secret.key", "secret")
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.ACCESS_KEY_ID)
+        .doesNotContainKey(
+            "iceberg.catalog.ice01." + IcebergVendedCredentialUtil.SECRET_ACCESS_KEY)
+        .satisfies(map ->
+            assertThat(map.get(InputFormatConfig.VENDED_STORAGE_CREDENTIALS))
+                .isNotBlank());

Review Comment:
   Good catch, added an assertion that `prefix()` remains `s3://my-bucket/` in the deserialized blob, along with the overridden endpoint and vended keys in `config()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "difin (via GitHub)" <gi...@apache.org>.
difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config, String name) {
       table = readTableObjectFromFile(location, config);
     }
     checkAndSetIoConfig(config, table);
+    IcebergVendedCredentialUtil.applyFromJobConf(table, config);

Review Comment:
   I tested in the Gravitino q-test: you are correct — vended credentials in `table.io().properties()` survive ser/de to executors. We still need to extract those credentials and stage them in Hive job configuration for two reasons:
   
   1. **Hadoop S3A.** Iceberg FileIO and Hadoop S3A do not share state; S3A only reads Configuration, and Tez/LLAP use S3A for `s3://` paths. At compile time (`configureInputJobProperties` / `configureInputJobCredentials`) we have `TableDesc`, not the final task `JobConf`, so vended credentials are staged in `jobProperties` and `jobSecrets` per HIVE-20651. At job submit, those maps are merged into `JobConf` / Hadoop Credentials and sent to executors before tasks run.
   
   2. **Iceberg FileIO endpoint override.** `applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed when REST catalogs vend S3 connection settings for their own network while Hive runs elsewhere. In the Gravitino q-test, Gravitino (in Docker) vends `s3.endpoint`=`http://minio:9000`, which tasks on the host cannot reach; session config sets a reachable endpoint (`http://<host>:9000`). `applyFromJobConf` takes vended credentials (from job conf or `FileIO.properties()`), replaces only endpoint and path-style with session values, leaves access keys unchanged, and installs the result on the FileIO via `setCredentials()`. In my debug, deserialized `table.io().properties()` still had `http://minio:9000`, so this step is required even when vended keys survive ser/de.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4848187367

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29359: Support Credential Vending in Hive Iceberg REST Catalog Client [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6474:
URL: https://github.com/apache/hive/pull/6474#issuecomment-4550781517

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [30 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6474&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6474&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6474&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6474)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org