You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "okumin (via GitHub)" <gi...@apache.org> on 2025/06/14 10:37:47 UTC
[PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
okumin opened a new pull request, #5870:
URL: https://github.com/apache/hive/pull/5870
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2158956762
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
I'll try to create a draft with the proposal, so we can collaborate more effectively.
You might be right and we should consider the possibility of supporting multiple endpoints, each with its own authentication. I haven't thought much about it.
FYI, we are using JWT downstream to enable Snowflake integration with HMS Iceberg RestCatalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152234056
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
ok, how about
````
String authType = MetastoreConf.getVar(configuration, ConfVars.THRIFT_METASTORE_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(
AuthType.JWT.name().equalsIgnoreCase(authType) ? AuthType.JWT : AuthType.SIMPLE,
configuration);
````
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
ok, how about
````
String authType = MetastoreConf.getVar(configuration, ConfVars.THRIFT_METASTORE_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(
AuthType.JWT.name().equalsIgnoreCase(authType) ? AuthType.JWT : AuthType.SIMPLE,
configuration);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146966468
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthJWT.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import com.nimbusds.jose.JWSAlgorithm;
+import com.nimbusds.jose.JWSHeader;
+import com.nimbusds.jose.JWSSigner;
+import com.nimbusds.jose.crypto.RSASSASigner;
+import com.nimbusds.jose.jwk.RSAKey;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.jwt.SignedJWT;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.rest.requests.CreateTableRequest;
+import org.apache.iceberg.rest.requests.RenameTableRequest;
+import org.apache.iceberg.types.Types;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.ok;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+public class TestHMSCatalogAuthJWT extends HMSTestBase {
+ private static final File JWT_AUTHKEY_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-authorized-key.json");
+ private static final File JWT_NOAUTHKEY_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-unauthorized-key.json");
+ private static final File JWT_JWKS_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-verification-jwks.json");
+ private static final int MOCK_JWKS_SERVER_PORT = 8089;
+
+ @ClassRule
+ public static final WireMockRule MOCK_JWKS_SERVER = new WireMockRule(MOCK_JWKS_SERVER_PORT);
+
+ private String generateJWT() throws Exception {
+ return generateJWT(JWT_AUTHKEY_FILE.toPath());
+ }
+
+ private String generateJWT(Path path) throws Exception {
+ return generateJWT(USER_1, path, TimeUnit.MINUTES.toMillis(5));
+ }
+
+ private static String generateJWT(String user, Path keyFile, long lifeTimeMillis) throws Exception {
+ RSAKey rsaKeyPair = RSAKey.parse(new String(java.nio.file.Files.readAllBytes(keyFile), StandardCharsets.UTF_8));
+ // Create RSA-signer with the private key
+ JWSSigner signer = new RSASSASigner(rsaKeyPair);
+ JWSHeader header = new JWSHeader
+ .Builder(JWSAlgorithm.RS256)
+ .keyID(rsaKeyPair.getKeyID())
+ .build();
+ Date now = new Date();
+ Date expirationTime = new Date(now.getTime() + lifeTimeMillis);
+ JWTClaimsSet claimsSet = new JWTClaimsSet.Builder()
+ .jwtID(UUID.randomUUID().toString())
+ .issueTime(now)
+ .issuer("auth-server")
+ .subject(user)
+ .expirationTime(expirationTime)
+ .claim("custom-claim-or-payload", "custom-claim-or-payload")
+ .build();
+ SignedJWT signedJWT = new SignedJWT(header, claimsSet);
+ // Compute the RSA signature
+ signedJWT.sign(signer);
+ return signedJWT.serialize();
+ }
+
+ private static Object clientCall(String jwt, URL url, String method, Object arg) throws IOException {
+ return clientCall(jwt, url, method, true, arg);
+ }
+
+ private static Object clientCall(String jwt, URL url, String method, boolean json, Object arg) throws IOException {
+ return clientCall(url, method, json, Collections.singletonMap("Authorization", "Bearer " + jwt), arg);
+ }
+
+ @Before
+ @Override
+ public void setUp() throws Exception {
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "jwt");
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.THRIFT_METASTORE_AUTHENTICATION_JWT_JWKS_URL,
+ "http://localhost:" + MOCK_JWKS_SERVER_PORT + "/jwks");
+ MOCK_JWKS_SERVER.stubFor(get("/jwks")
+ .willReturn(ok()
+ .withBody(Files.readAllBytes(JWT_JWKS_FILE.toPath()))));
+ super.setUp();
Review Comment:
can we pass `conf` in `setUp()` for more clarify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152184347
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
This line configures [an HTTP server for HiveMetaStore with Thrift over HTTP](https://github.com/apache/hive/blob/174ff56b77b212bf51ee01587de9cf22e77f2dd3/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L966-L977), not for Iceberg REST or Properties REST. This change keeps the original behavior. We can't fully encode `hive.metastore.authentication` there, as it doesn't have SIMPLE.
As for the necessity of consolidation, I am currently biased toward separation. Let's say we introduced OAuth 2.0 with OIDC for the Iceberg REST catalog, and a user wanted to use it. Can all PROPERTIES clients immediately use OAuth? I am a little skeptical that different HTTP endpoints can share the same authentication method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sence?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2996462992
@deniskuzZ I rebased it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #5870:
URL: https://github.com/apache/hive/pull/5870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146979777
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
can we update the [HMS config in docker](https://github.com/apache/hive/blob/2826a6e6dfaa5ff3c0f3e077b55b2994c98f60c4/standalone-metastore/metastore-server/src/docker/conf/metastore-site.xml#L41) to NONE as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified Rest API.
UnityCatalog already support a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
@okumin meanwhile could we please do the following refactor using parametrized test and RESTCatalog client
````
@Parameters(name = "authType = {0}")
public static List<Object> authType() {
return Arrays.asList(AuthType.JWT, AuthType.NONE);
}
@Parameter(0)
public ServletSecurity.AuthType authType;
protected RESTCatalog restCatalog;
````
in setUp (use RESTCatalog client instead of manually constructed http requests)
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
restCatalog = initCatalog(catalog.name());
}
private RESTCatalog initCatalog(String catalogName) throws Exception {
URI catalogUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
RESTCatalog catalog = new RESTCatalog();
Map<String, String> properties = Maps.newHashMap();
properties.put(CatalogProperties.URI, catalogUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
catalog.initialize(catalogName, properties);
return catalog;
}
````
refactored Test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.create();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
HMS RestCatalog would proxy the metadata request to the HMS or one of the registered external catalogs using the stored connection details and credentials.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2995455966
hi @okumin, could you please rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | SIMPLE, JWT | Hive, etc. |
| HMS (REST) | NONE, SIMPLE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2996071454
I probably think the consensus is that this is not a blocker, but I try to rebase this pr promptly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146974819
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -156,6 +160,10 @@ public String getServletInfo() {
* @return a servlet instance or null if security initialization fails
*/
public HttpServlet proxy(HttpServlet servlet) {
+ if (authType == AuthType.NONE) {
+ LOG.info("No security enabled, not proxying servlet {}", servlet.toString());
Review Comment:
minor: `Security is disabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146965612
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthJWT.java:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.rest;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import com.nimbusds.jose.JWSAlgorithm;
+import com.nimbusds.jose.JWSHeader;
+import com.nimbusds.jose.JWSSigner;
+import com.nimbusds.jose.crypto.RSASSASigner;
+import com.nimbusds.jose.jwk.RSAKey;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.jwt.SignedJWT;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.rest.requests.CreateTableRequest;
+import org.apache.iceberg.rest.requests.RenameTableRequest;
+import org.apache.iceberg.types.Types;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.ok;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+public class TestHMSCatalogAuthJWT extends HMSTestBase {
+ private static final File JWT_AUTHKEY_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-authorized-key.json");
+ private static final File JWT_NOAUTHKEY_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-unauthorized-key.json");
+ private static final File JWT_JWKS_FILE =
+ new File(BASE_DIR,"src/test/resources/auth/jwt/jwt-verification-jwks.json");
+ private static final int MOCK_JWKS_SERVER_PORT = 8089;
+
+ @ClassRule
+ public static final WireMockRule MOCK_JWKS_SERVER = new WireMockRule(MOCK_JWKS_SERVER_PORT);
+
+ private String generateJWT() throws Exception {
+ return generateJWT(JWT_AUTHKEY_FILE.toPath());
+ }
+
+ private String generateJWT(Path path) throws Exception {
+ return generateJWT(USER_1, path, TimeUnit.MINUTES.toMillis(5));
+ }
+
+ private static String generateJWT(String user, Path keyFile, long lifeTimeMillis) throws Exception {
+ RSAKey rsaKeyPair = RSAKey.parse(new String(java.nio.file.Files.readAllBytes(keyFile), StandardCharsets.UTF_8));
+ // Create RSA-signer with the private key
+ JWSSigner signer = new RSASSASigner(rsaKeyPair);
+ JWSHeader header = new JWSHeader
+ .Builder(JWSAlgorithm.RS256)
+ .keyID(rsaKeyPair.getKeyID())
+ .build();
+ Date now = new Date();
+ Date expirationTime = new Date(now.getTime() + lifeTimeMillis);
+ JWTClaimsSet claimsSet = new JWTClaimsSet.Builder()
+ .jwtID(UUID.randomUUID().toString())
+ .issueTime(now)
+ .issuer("auth-server")
+ .subject(user)
+ .expirationTime(expirationTime)
+ .claim("custom-claim-or-payload", "custom-claim-or-payload")
+ .build();
+ SignedJWT signedJWT = new SignedJWT(header, claimsSet);
+ // Compute the RSA signature
+ signedJWT.sign(signer);
+ return signedJWT.serialize();
+ }
+
+ private static Object clientCall(String jwt, URL url, String method, Object arg) throws IOException {
+ return clientCall(jwt, url, method, true, arg);
+ }
+
+ private static Object clientCall(String jwt, URL url, String method, boolean json, Object arg) throws IOException {
+ return clientCall(url, method, json, Collections.singletonMap("Authorization", "Bearer " + jwt), arg);
+ }
+
+ @Before
+ @Override
+ public void setUp() throws Exception {
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "jwt");
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.THRIFT_METASTORE_AUTHENTICATION_JWT_JWKS_URL,
+ "http://localhost:" + MOCK_JWKS_SERVER_PORT + "/jwks");
+ MOCK_JWKS_SERVER.stubFor(get("/jwks")
+ .willReturn(ok()
+ .withBody(Files.readAllBytes(JWT_JWKS_FILE.toPath()))));
+ super.setUp();
+ }
+
+ @After
+ @Override
+ public void tearDown() throws Exception {
+ super.tearDown();
Review Comment:
why override?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146972688
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -78,33 +78,37 @@
* </p>
*/
public class ServletSecurity {
+ public enum AuthType {
+ NONE, SIMPLE, JWT;
+
+ public static AuthType fromString(String type) {
+ return AuthType.valueOf(type.toUpperCase());
+ }
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(ServletSecurity.class);
static final String X_USER = MetaStoreUtils.USER_NAME_HTTP_HEADER;
private final boolean isSecurityEnabled;
- private final boolean jwtAuthEnabled;
+ private final AuthType authType;
private final Configuration conf;
private JWTValidator jwtValidator = null;
- public ServletSecurity(String authType, Configuration conf) {
- this(conf, isAuthJwt(authType));
- }
-
- public ServletSecurity(Configuration conf, boolean jwt) {
+ public ServletSecurity(AuthType authType, Configuration conf) {
this.conf = conf;
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
- this.jwtAuthEnabled = jwt;
+ this.authType = authType;
Review Comment:
why not do `this.authType = AuthType.fromString(authType)` here and accept the string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`METASTORE_REST_CATALOG_AUTHENTICATION("metastore.rest-catalog.authentication"`
````
String authType = MetastoreConf.getVar(configuration, ConfVars.METASTORE_REST_CATALOG_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146972688
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -78,33 +78,37 @@
* </p>
*/
public class ServletSecurity {
+ public enum AuthType {
+ NONE, SIMPLE, JWT;
+
+ public static AuthType fromString(String type) {
+ return AuthType.valueOf(type.toUpperCase());
+ }
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(ServletSecurity.class);
static final String X_USER = MetaStoreUtils.USER_NAME_HTTP_HEADER;
private final boolean isSecurityEnabled;
- private final boolean jwtAuthEnabled;
+ private final AuthType authType;
private final Configuration conf;
private JWTValidator jwtValidator = null;
- public ServletSecurity(String authType, Configuration conf) {
- this(conf, isAuthJwt(authType));
- }
-
- public ServletSecurity(Configuration conf, boolean jwt) {
+ public ServletSecurity(AuthType authType, Configuration conf) {
this.conf = conf;
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
- this.jwtAuthEnabled = jwt;
+ this.authType = authType;
Review Comment:
why not do `this.authType = AuthType.fromString(authType)` here and accept the string?
you won't need the hack like `this(jwt ? AuthType.JWT : AuthType.SIMPLE, conf);` in the 2nd constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146974819
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -156,6 +160,10 @@ public String getServletInfo() {
* @return a servlet instance or null if security initialization fails
*/
public HttpServlet proxy(HttpServlet servlet) {
+ if (authType == AuthType.NONE) {
+ LOG.info("No security enabled, not proxying servlet {}", servlet.toString());
Review Comment:
minor: `Servlet security is disabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147410150
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
I think we should separate one at least for Iceberg. OAuth2 is not used in Hive's endpoints, but the Iceberg community prefers it. Another table format might prefer SPNEGO or something else if we support it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
meanwhile we could easily parametrize it
````
protected ServletSecurity.AuthType authType; - JWT & NONE
protected RESTCatalog restCatalog;
````
in setUp (use RESTCatalog client instead of manually constructed http requests)
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
URI iceUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
Map<String, String> properties = Maps.newHashMap();
properties.put("type", "rest");
properties.put("uri", iceUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
final String catalogName = catalog.name();
restCatalog = (RESTCatalog) CatalogUtil.buildIcebergCatalog(catalogName, properties, new Configuration());
restCatalog.initialize(catalogName, properties);
````
generic test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.createTransaction()
.commitTransaction();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
meanwhile we could easily parametrize it
````
protected ServletSecurity.AuthType authType; - JWT & NONE
protected RESTCatalog restCatalog;
````
in setUp
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
URI iceUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
Map<String, String> properties = Maps.newHashMap();
properties.put("type", "rest");
properties.put("uri", iceUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
final String catalogName = catalog.name();
restCatalog = (RESTCatalog) CatalogUtil.buildIcebergCatalog(catalogName, properties, new Configuration());
restCatalog.initialize(catalogName, properties);
````
generic test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.createTransaction()
.commitTransaction();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`METASTORE_REST_CATALOG_AUTH("metastore.rest-catalog.authentication"`
````
String authType = MetastoreConf.getVar(configuration, ConfVars.METASTORE_REST_CATALOG_AUTH);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146967298
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
Review Comment:
same, no need to override
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146974819
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -156,6 +160,10 @@ public String getServletInfo() {
* @return a servlet instance or null if security initialization fails
*/
public HttpServlet proxy(HttpServlet servlet) {
+ if (authType == AuthType.NONE) {
+ LOG.info("No security enabled, not proxying servlet {}", servlet.toString());
Review Comment:
minor: `LOG.warn("Servlet security is disabled {} , servlet.toString());`
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -156,6 +160,10 @@ public String getServletInfo() {
* @return a servlet instance or null if security initialization fails
*/
public HttpServlet proxy(HttpServlet servlet) {
+ if (authType == AuthType.NONE) {
+ LOG.info("No security enabled, not proxying servlet {}", servlet.toString());
Review Comment:
minor: `LOG.warn("Servlet security is disabled {}, servlet.toString());`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147013121
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
Yes. I'm thinking of replacing all tests with the compatibility kit before applying cosmetic changes on this test cases. Basically, the current coverage is too small.
https://issues.apache.org/jira/browse/HIVE-29019
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
OK. I am considering if we should support SIMPLE and JWT. They are likely inherited from `hive.metastore.properties.servlet.auth`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147021122
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
That comes from `PROPERTIES_SERVLET_AUTH` - another servlet, but I don't understand why can't we have 1 generic config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2973284578
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2158186754
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
It was a very good discussion. This discussion involved the further development direction and positioning of HMS in the future.
First, I think we should write a detailed proposal and discuss it clearly in the community email. Because I feel that many people are not clear about the positioning of the HMS rest catalog, which may cause various misunderstandings.
Returning to the current issue, I basically agree with Denys' statement that HMS(Rest) in the future will be similar to the Unity Catalog, capable of serving as an Iceberg rest catalog server (its primary function), and also able to register other catalogs (such as Hive tables from other HMS).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS RestCatalog (Iceberg, etc..) clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | SIMPLE, JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | SIMPLE, JWT | Hive, etc. |
| HMS (REST) | NONE, SIMPLE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2160221824
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
This is a good discussion, but I started thinking it does not need to be a blocker to make the current Iceberg REST API minimally viable. I separated the debate in the following ticket.
https://issues.apache.org/jira/browse/HIVE-29038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155928298
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
I'd like to follow your take if you have such a context. I'd be more than happy if you had the summary of the vision :) > it's intended for use by the HMS RestCatalog (aka UnityCatalog) serving federated metadata (including Hive external and ACID tables)
Anyway, I would like the conclusion here. This is the summary I have.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| Properties API | SIMPLE, JWT | (Who uses it?) |
| Iceberg REST | NONE, OAuth 2.0 | Icebrerg clients |
| UnityCatalog | OAuth 2.0? | Databricks, etc. |
What should we support in `metastore.rest-catalog.authentication`? Having the union is likely a conclusion. I'm concerned that some clients can't support an arbitrary protocol because some of them are out of our control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
meanwhile we could easily parametrize it
````
protected ServletSecurity.AuthType authType; - JWT & NONE
protected RESTCatalog restCatalog;
````
in setUp (use RESTCatalog client instead of manually constructed http requests)
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
restCatalog = initCatalog(catalog.name());
}
private RESTCatalog initCatalog(String catalogName) throws Exception {
URI catalogUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
RESTCatalog catalog = new RESTCatalog();
Map<String, String> properties = Maps.newHashMap();
properties.put(CatalogProperties.URI, catalogUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
catalog.initialize(catalogName, properties);
return catalog;
}
````
generic test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.create();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
HMS RestCatalog would proxy the metadata request to the HMSHandler or one of the registered external catalogs using the stored connection details and credentials.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2972644673
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146969518
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
is that the only difference?
if yes, can't we move the tests into base class and override just the client function call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146972688
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -78,33 +78,37 @@
* </p>
*/
public class ServletSecurity {
+ public enum AuthType {
+ NONE, SIMPLE, JWT;
+
+ public static AuthType fromString(String type) {
+ return AuthType.valueOf(type.toUpperCase());
+ }
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(ServletSecurity.class);
static final String X_USER = MetaStoreUtils.USER_NAME_HTTP_HEADER;
private final boolean isSecurityEnabled;
- private final boolean jwtAuthEnabled;
+ private final AuthType authType;
private final Configuration conf;
private JWTValidator jwtValidator = null;
- public ServletSecurity(String authType, Configuration conf) {
- this(conf, isAuthJwt(authType));
- }
-
- public ServletSecurity(Configuration conf, boolean jwt) {
+ public ServletSecurity(AuthType authType, Configuration conf) {
this.conf = conf;
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
- this.jwtAuthEnabled = jwt;
+ this.authType = authType;
Review Comment:
why not do `this.authType = AuthType.fromString(authType)` here and accept the string?
you won't need the hack like `this(jwt ? AuthType.JWT : AuthType.SIMPLE, conf);` in the 2nd constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`METASTORE_REST_CATALOG_AUTH("metastore.rest-catalog.authentication"` since these servlets are deployed within the same servlet container.
````
String authType = MetastoreConf.getVar(configuration, ConfVars.METASTORE_REST_CATALOG_AUTH);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152184347
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
This line configures [an HTTP server for HiveMetaStore with Thrift over HTTP](https://github.com/apache/hive/blob/174ff56b77b212bf51ee01587de9cf22e77f2dd3/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L966-L977), not for Iceberg REST or Properties REST. This change keeps the original behavior. We can't fully encode `hive.metastore.authentication` there, as it doesn't have SIMPLE.
As for the necessity of consolidation, I am currently biased toward separation. Let's say we introduced OAuth 2.0 with OIDC for the Iceberg REST catalog, and a user wanted to use it. Can all PROPERTIES clients immediately use OAuth? I am a little skeptical that different HTTP endpoints can share the same authentication method even if they are on the same machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147410150
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
I think we should separate one at least for Iceberg. OAuth2 is not used in Hive's endpoints, but the Iceberg community prefers it. Another table format might prefer SPNEGO or something else if we support it. Anyway, a set of auth methods relies on the product, the community, and the client implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`METASTOTE_REST_CATALOG_AUTHENTICATION("metastore.rest-catalog.authentication"`
````
String authType = MetastoreConf.getVar(configuration, ConfVars.METASTOTE_REST_CATALOG_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | REST (Iceberg, etc..) clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | SIMPLE, JWT, etc | Hive, etc. |
| HMS (REST) | NONE, SIMPLE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2160263910
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
Yes, 100%. That's beyond the scope of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2997349568
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS(HTTP) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS clients |
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146972688
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -78,33 +78,37 @@
* </p>
*/
public class ServletSecurity {
+ public enum AuthType {
+ NONE, SIMPLE, JWT;
+
+ public static AuthType fromString(String type) {
+ return AuthType.valueOf(type.toUpperCase());
+ }
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(ServletSecurity.class);
static final String X_USER = MetaStoreUtils.USER_NAME_HTTP_HEADER;
private final boolean isSecurityEnabled;
- private final boolean jwtAuthEnabled;
+ private final AuthType authType;
private final Configuration conf;
private JWTValidator jwtValidator = null;
- public ServletSecurity(String authType, Configuration conf) {
- this(conf, isAuthJwt(authType));
- }
-
- public ServletSecurity(Configuration conf, boolean jwt) {
+ public ServletSecurity(AuthType authType, Configuration conf) {
this.conf = conf;
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
- this.jwtAuthEnabled = jwt;
+ this.authType = authType;
Review Comment:
why not do `this.authType = AuthType.fromString(authType)` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147021122
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
That comes from `PROPERTIES_SERVLET_AUTH` - another servlet that has 0 usages, but I don't understand why can't we have 1 generic config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2158831458
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
@deniskuzZ @zhangbutao Thanks. I also think we should discuss the excellent vision on the mailing list. I want to support it. However, one point remains unclear to me.
Unity Catalog provides two types of endpoints: One is Unity REST API(`/api/2.1/unity-catalog`) and the other is Iceberg REST API(`/api/2.1/unity-catalog/iceberg-rest`). [Unity REST API is likely to authenticate a user with Personal Access Token](https://docs.databricks.com/aws/en/external-access/unity-rest). [Iceberg REST API is likely to authenticate a user with OAuth or PAT](https://docs.databricks.com/aws/en/external-access/iceberg). I understand we want the equivalent of the Unity REST; Let's call it HMS v2 REST.
I don't understand why we can use the consistent auth across the endpoints. HMS v2 REST clients will be able to use SIMPLE(i.e., a client sends a user name via `x-actor-username` header) if we implement the client so. However, as a reality, [most Iceberg REST clients probably do not support it](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/AuthManagers.java). I think Trino has no method to use SIMPLE and JWT.
If I were saying something strange, I would follow the recommendation now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2160222495
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
It's okay if it's a blocker, but I'm removing the current test case in this pull request. I'll do it if we want to merge HIVE-28998 first, and the change is a must.
https://github.com/apache/hive/pull/5887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
@okumin meanwhile could we please do the following refactor parametrize it
````
@Parameters(name = "authType = {0}")
public static List<Object> authType() {
return Arrays.asList(AuthType.JWT, AuthType.NONE);
}
@Parameter(0)
public ServletSecurity.AuthType authType;
protected RESTCatalog restCatalog;
````
in setUp (use RESTCatalog client instead of manually constructed http requests)
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
restCatalog = initCatalog(catalog.name());
}
private RESTCatalog initCatalog(String catalogName) throws Exception {
URI catalogUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
RESTCatalog catalog = new RESTCatalog();
Map<String, String> properties = Maps.newHashMap();
properties.put(CatalogProperties.URI, catalogUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
catalog.initialize(catalogName, properties);
return catalog;
}
````
refactored Test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.create();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152273388
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
`hive.metastore.iceberg.catalog.servlet.auth` config rename has caused confusion. It's not meant to be specific to Iceberg - it's intended for use by the HMS RestCatalog (aka UnityCatalog) serving federated metadata (including Hive external and ACID tables)."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`METASTORE_REST_CATALOG_AUTHENTICATION("metastore.rest-catalog.authentication"`
````
String authType = MetastoreConf.getVar(configuration, ConfVars.METASTORE_REST_CATALOG_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2146974394
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java:
##########
@@ -78,33 +78,37 @@
* </p>
*/
public class ServletSecurity {
+ public enum AuthType {
+ NONE, SIMPLE, JWT;
+
+ public static AuthType fromString(String type) {
+ return AuthType.valueOf(type.toUpperCase());
+ }
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(ServletSecurity.class);
static final String X_USER = MetaStoreUtils.USER_NAME_HTTP_HEADER;
private final boolean isSecurityEnabled;
- private final boolean jwtAuthEnabled;
+ private final AuthType authType;
private final Configuration conf;
private JWTValidator jwtValidator = null;
- public ServletSecurity(String authType, Configuration conf) {
- this(conf, isAuthJwt(authType));
- }
-
- public ServletSecurity(Configuration conf, boolean jwt) {
+ public ServletSecurity(AuthType authType, Configuration conf) {
this.conf = conf;
this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
- this.jwtAuthEnabled = jwt;
+ this.authType = authType;
}
- private static boolean isAuthJwt(String authType) {
- return "jwt".equalsIgnoreCase(authType);
+ public ServletSecurity(Configuration conf, boolean jwt) {
+ this(jwt ? AuthType.JWT : AuthType.SIMPLE, conf);
Review Comment:
this looks confusing, can we get rid of this constructor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147021122
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
That comes from `PROPERTIES_SERVLET_AUTH` - another servlet that has 0 usages. I don't understand why we can't have 1 generic config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2149481104
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
why do we need these hacks? can we drop the jwt var completely?
I think we should replace `ICEBERG_CATALOG_SERVLET_AUTH` & `PROPERTIES_SERVLET_AUTH`
with
`REST_CATALOG_AUTHENTICATION("metastore.rest-catalog.authentication"`
````
String authType = MetastoreConf.getVar(configuration, ConfVars.REST_CATALOG_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(AuthType.fromString(authType), configuration);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5870:
URL: https://github.com/apache/hive/pull/5870#issuecomment-2973475564
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5870&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5870&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5870&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5870)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2147410150
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
I think we have separate one at least for Iceberg. OAuth2 is not used in Hive's endpoints, but the Iceberg community prefers it. Another table format might prefer SPNEGO or something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152234056
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
ok, how about
````
String authType = MetastoreConf.getVar(configuration, ConfVars.THRIFT_METASTORE_AUTHENTICATION);
ServletSecurity security = new ServletSecurity(
AuthType.JWT.name().equalsIgnoreCase(authType) ? AuthType.JWT : AuthType.SIMPLE,
configuration);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2152247296
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
i find it's bizarre to support different authentication methods for multiple endpoints served by the same servlet in a production cluster
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -431,7 +432,8 @@ public void setThreadFactory(ThreadFactory threadFactory) {
processor = new ThriftHiveMetastore.Processor<>(handler);
LOG.info("Starting DB backed MetaStore Server with generic processor");
boolean jwt = MetastoreConf.getVar(conf, ConfVars.THRIFT_METASTORE_AUTHENTICATION).equalsIgnoreCase("jwt");
- ServletSecurity security = new ServletSecurity(conf, jwt);
+ AuthType authType = jwt ? AuthType.JWT : AuthType.SIMPLE;
Review Comment:
i find it bizarre to support different authentication methods for multiple endpoints served by the same servlet in a production cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2158831458
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
@deniskuzZ @zhangbutao Thanks. I also think we should discuss the excellent vision on the mailing list. I want to support it. However, one point remains unclear to me.
Unity Catalog provides two types of endpoints: One is Unity REST API(`/api/2.1/unity-catalog`) and the other is Iceberg REST API(`/api/2.1/unity-catalog/iceberg-rest`). [Unity REST API is likely to authenticate a user with Personal Access Token](https://docs.databricks.com/aws/en/external-access/unity-rest). [Iceberg REST API is likely to authenticate a user with OAuth or PAT](https://docs.databricks.com/aws/en/external-access/iceberg). I understand we want the equivalent of the Unity REST; Let's call it HMS v2 REST.
I don't understand why we can use the consistent auth across the endpoints. HMS v2 REST clients will be able to use SIMPLE(i.e., a client sends a user name via `x-actor-username` header) if we implement the client so. However, as a reality, [most Iceberg REST clients probably do not support it](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/AuthManagers.java).
If I were saying something strange, I would follow the recommendation now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | Rest clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2155329314
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
meanwhile we could easily parametrize it
````
protected ServletSecurity.AuthType authType; - JWT & NONE
protected RESTCatalog restCatalog;
````
in setUp (use RESTCatalog client instead of manually constructed http requests)
````
MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, authType.name());
URI iceUri = URI.create("http://localhost:" + catalogPort + "/" + catalogPath);
Map<String, String> properties = Maps.newHashMap();
properties.put("type", "rest");
properties.put("uri", iceUri.toString());
if (authType == ServletSecurity.AuthType.JWT) {
properties.put("token", generateJWT());
}
final String catalogName = catalog.name();
restCatalog = (RESTCatalog) CatalogUtil.buildIcebergCatalog(catalogName, properties, new Configuration());
restCatalog.initialize(catalogName, properties);
````
generic test snippet
````
@Test
public void testTableAPI() throws Exception {
Schema schema = getTestSchema();
final String tblName = "tbl_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier TBL = TableIdentifier.of(DB_NAME, tblName);
String location = temp.newFolder(TBL.toString()).toString();
restCatalog
.buildTable(TBL, schema)
.withLocation(location)
.create();
Table table = catalog.loadTable(TBL);
Assert.assertEquals(location, table.location());
// rename table
final String rtblName = "TBL_" + Integer.toHexString(RND.nextInt(65536));
final TableIdentifier rtableIdent = TableIdentifier.of(DB_NAME, rtblName);
restCatalog.renameTable(TBL, rtableIdent);
table = catalog.loadTable(rtableIdent);
Assert.assertEquals(
String.join(".", catalog.name(),
rtableIdent.toString()), table.name());
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | SIMPLE, JWT | Hive, etc. |
| HMS (REST) | NONE, SIMPLE, JWT, OAuth 2.0 | HMS RestCatalog clients (Iceberg, etc..) |
The HMS RestCatalog would handle the proxying of metadata requests to either the HMSHandler, Iceberg's HiveCatalog or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | HMS REST (Iceberg, etc..) clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2156625705
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1867,7 +1867,7 @@ public enum ConfVars {
" positive value will be used as-is."
),
ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth",
- "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"),
+ "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"),
Review Comment:
hi @okumin,
The idea was to expose the metadata management functionality via REST in HMS.
Currently we are focused only on exposing iceberg tables, however, we could extend it with the support of Hive managed format like ACID.
Another thing is federation. Register multiple catalog connectors in HMS (keep connection details with credentials in HMS DB) and expose it via same Unified/Iceberg Rest API.
UnityCatalog already supports a Managed catalog and ability to integrate with the Foreign catalogs like Glue, HMS, Iceberg Rest Catalogs.
| Service | AuthN supported by clients | users |
|-|-|-|
| HMS (HTTP over Thrift) | NONE(probably, like a SIMPLE), JWT | Hive, etc. |
| HMS (REST) | NONE, JWT, OAuth 2.0 | Rest (Iceberg, etc..) clients |
The HMS RestCatalog would handle proxying metadata requests to either the HMSHandler or one of the registered external catalogs, using the stored connection details and credentials.
@okumin, does it make sense?
As far as I know, the Properties API isn't used by anyone. That was one of my concerns, given the lack of a clear vision or usage patterns. I felt like it's yet another key-value store bicycle without a defined purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28998: Support Iceberg REST Catalog without authentication [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5870:
URL: https://github.com/apache/hive/pull/5870#discussion_r2160263228
##########
standalone-metastore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java:
##########
@@ -36,45 +39,41 @@
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.types.Types;
-import static org.apache.iceberg.types.Types.NestedField.required;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestHMSCatalog extends HMSTestBase {
- public TestHMSCatalog() {
- super();
- }
-
+public class TestHMSCatalogAuthNone extends HMSTestBase {
@Before
@Override
public void setUp() throws Exception {
- super.setUp();
+ conf = MetastoreConf.newMetastoreConf();
+ MetastoreConf.setVar(conf, MetastoreConf.ConfVars.ICEBERG_CATALOG_SERVLET_AUTH, "none");
+ super.setUp();
}
-
+
@After
@Override
public void tearDown() throws Exception {
- super.tearDown();
+ super.tearDown();
}
@Test
public void testCreateNamespaceHttp() throws Exception {
String ns = "nstesthttp";
// list namespaces
URL url = new URL("http://hive@localhost:" + catalogPort + "/"+catalogPath+"/v1/namespaces");
- String jwt = generateJWT();
// check namespaces list (ie 0)
- Object response = clientCall(jwt, url, "GET", null);
+ Object response = clientCall(url, "GET", Collections.emptyMap(), null);
Review Comment:
Let's merge https://github.com/apache/hive/pull/5887 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org