You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "zhangbutao (via GitHub)" <gi...@apache.org> on 2026/01/14 09:34:45 UTC
[PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
zhangbutao opened a new pull request, #6267:
URL: https://github.com/apache/hive/pull/6267
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4126289430
> This could make some confusing. For example, we declare a S3 catalog but the location is absent, then this catalog is created with a hdfs location, some databases without explicit location will be put under the hdfs, while some are on the cloud, it might be a problem if we use the FileSystem created by the catalog's properties to operate against these tables or databases.
>
> In HMS, we can enforce the location check when creating the catalog, if it's null and the catalog is native, then we should throw the exception.
I understand what you mean. I think what we're discussing is more about usage habits.
Before the catalog capability was available, Hive used databases as a way to isolate data, and when users created a Hive database, they could omit the location, in which case the database would use the default location. If users wanted to set a value different from the default catalog location, they could specify a specific location, such as an S3 location, when creating the catalog.
This syntax of omiting the location is more flexible, and I think users might be more accustomed to the usage of having a default catalog location.
I would also like to hear other folks' thoughts on the default catalog location. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type. I assume for GLUE, there would be "glue" type, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the warehousePath?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068029328
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
Review Comment:
could we do this in the var initialization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3141721429
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java:
##########
@@ -51,10 +52,9 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
@Override
public void analyzeInternal(ASTNode root) throws SemanticException {
Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));
Review Comment:
that's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6267: HIVE-29241: Distinguish the default location of the database by catalog
URL: https://github.com/apache/hive/pull/6267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type.
- native (local) - that's what we have now, with external, acid and iceberg tables support;
- jdbc,
- iceberg (jdbc/hive/rest),
- hive (hms, glue, etc),
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type. I assume for GLUE, there would be "glue" type, right?
- native (local) - that's what we have now, with external, acid and iceberg tables support;
- jdbc,
- iceberg (jdbc/hive/rest),
- hive,
- glue,
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | | External HMS |
| Hive tables | | Hive tables |
| Iceberg | | Iceberg tables |
| tables | +-----------------+
| | | REST Catalog |
| | | Iceberg tables |
+-------------+ +-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4200282209
hi @zhangbutao, is the PR ready for 2nd round or review and includes the changes we discussed here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3083885143
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -694,6 +694,9 @@ private void rollbackTransactionToSavePoint(String savePoint) {
@Override
public void createCatalog(Catalog cat) throws MetaException {
LOG.debug("Creating catalog {}", cat);
+
+ ensureCatalogType(cat);
Review Comment:
can we uplift the check to `HMSHandler#create_catalog`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068050601
##########
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:
##########
@@ -132,6 +132,8 @@ public void authorizeDbLevelOperations(Privilege[] readRequiredPriv, Privilege[]
Path root = null;
try {
initWh();
+ // TODO catalog. Need to determine auth root path based on catalog name.
Review Comment:
could you please add the ticket to the EPIC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068056716
##########
ql/src/test/queries/clientnegative/lockneg_try_lock_cat_db_in_use.q:
##########
@@ -1,7 +1,7 @@
set hive.lock.numretries=0;
set hive.support.concurrency=true;
-CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog';
+CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog' PROPERTIES('type'='native');
Review Comment:
is redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068065595
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+public class CatalogUtil {
+ // Catalog type constants
+ public static final String NATIVE = "native";
Review Comment:
no need for Strings, please use enum values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068071794
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +330,56 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
- if (inExternalWH) {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
- }
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
- } else {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
- }
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ Database db = new Database();
Review Comment:
are you using Database only as a DTO object? instead you could use a record CatalogDb as mentioned earlier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137315736
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +330,56 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
- if (inExternalWH) {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
- }
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
- } else {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
- }
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ Database db = new Database();
Review Comment:
CatalogDb is a self-describing, single-purpose DTO. It clearly communicates intent, keeps the data model focused, and avoids overloading more generic structures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-6216152506
@zhangbutao, is this completely abandoned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809561947
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, `NATIVE` means Hive only?
if we plan to support federation won't `REMOTE` be a better choice instead of `ICEBERG`? Iceberg is a table type
FYI: https://polaris.apache.org/in-dev/unreleased/federation/hive-metastore-federation/
@dengzhhu653, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3023613734
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
might be. i was just thinking that when you describe catalog - we need to add the flag "EXTERNAL" = {"true", "false"} or similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/table # non default catalog
/warehouse/<database_name>.db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068035384
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java:
##########
@@ -51,10 +52,9 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
@Override
public void analyzeInternal(ASTNode root) throws SemanticException {
Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));
Review Comment:
How about `record CatalogDb(String catalog, String database) {}`
please see https://github.com/apache/hive/pull/6379#discussion_r3045880254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2923394783
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
Review Comment:
I followed the previous method of defining parameters. Currently, it seems unnecessary to define parameters in HiveConf, as it's somewhat redundant.
Removed this parameter from HiveConf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | ---------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068045324
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:
##########
@@ -388,14 +389,34 @@ public static void createDbExportDump(FileSystem fs, Path metadataPath, Database
private static void updateIfCustomDbLocations(Database database, Configuration conf) throws SemanticException {
try {
- String whLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL,
- MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE));
- Path dbDerivedLoc = new Path(whLocatoion, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ String catName = database.getCatalogName();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equals(catName);
+
+ // external warehouse root
+ String whLocation = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL : MetastoreConf.ConfVars.WAREHOUSE_CATALOG_EXTERNAL,
+ MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE : MetastoreConf.ConfVars.WAREHOUSE_CATALOG));
+
+ if (!isDefaultCatalog) {
+ whLocation = new Path(whLocation, catName).toString();
+ }
+
+ Path dbDerivedLoc = new Path(whLocation, dbName + DATABASE_PATH_SUFFIX);
String defaultDbLoc = Utilities.getQualifiedPath((HiveConf) conf, dbDerivedLoc);
database.putToParameters(ReplConst.REPL_IS_CUSTOM_DB_LOC,
Boolean.toString(!defaultDbLoc.equals(database.getLocationUri())));
- String whManagedLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE);
- Path dbDerivedManagedLoc = new Path(whManagedLocatoion, database.getName().toLowerCase()
+
+ // managed warehouse root
+ String whManagedLocatoion = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE
Review Comment:
please see https://github.com/apache/hive/pull/6267#discussion_r3068019699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068048439
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractRepository.java:
##########
@@ -82,7 +85,11 @@ protected Database initDatabase(Hive hive) {
db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
if (db == null) {
LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
- String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database();
Review Comment:
looks a bit weird creating 2 Database objects in same method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4231620526
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [75 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [1.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
I'm aligned on this as well
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
I'm aligned on this as well
+1
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
i am on same page here
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-6413351494
> @zhangbutao, is this completely abandoned?
@deniskuzZ Will update this PR soon. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2689761412
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -56,6 +56,7 @@
import org.apache.hadoop.hive.metastore.api.Partition;
import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
import org.apache.hadoop.hive.metastore.api.Table;
+import org.stringtemplate.v4.ST;
Review Comment:
The import `org.stringtemplate.v4.ST` is added but never used in the visible code changes. This appears to be an unused import that should be removed.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +72,23 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ assert props != null;
Review Comment:
The assertion `assert props != null;` on line 75 will not throw a user-friendly error if assertions are disabled (which is common in production). Since the grammar now requires PROPERTIES (line 1136 in HiveParser.g), `props` should never be null. However, if it is somehow null, this should throw a proper SemanticException instead of relying on an assertion. Consider replacing with a proper null check and exception.
```suggestion
if (props == null) {
throw new SemanticException("PROPERTIES clause is required for CREATE CATALOG");
}
```
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -156,33 +170,67 @@ public Path getDnsPath(Path path) throws MetaException {
* This involves opening the FileSystem corresponding to the warehouse root
* dir (but that should be ok given that this is only called during DDL
* statements for non-external tables).
+ *
+ * @deprecated use {@link #getWhRoot(String)}
*/
public Path getWhRoot() throws MetaException {
- if (whRoot != null) {
- return whRoot;
+ return getWhRoot(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRoot(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName.trim());
+
+ Path rootDir = isDefault ? whRoot : whCatRoot;
+ if (rootDir == null) {
+ rootDir = getDnsPath(new Path(isDefault ? whRootString : whCatRootString));
+ if (isDefault) {
+ whRoot = rootDir;
+ } else {
+ whCatRoot = rootDir;
+ }
}
- whRoot = getDnsPath(new Path(whRootString));
- return whRoot;
+
+ return isDefault ? rootDir : new Path(rootDir, catalogName);
Review Comment:
The caching logic for `whCatRoot` is incorrect. When a non-default catalog is requested, the code caches the base path (from `whCatRootString`) in `whCatRoot`, but then returns `new Path(rootDir, catalogName)` which includes the catalog name. On subsequent calls, it retrieves the cached path and adds the catalog name again, resulting in paths like `/base/catalog1/catalog1`. The cache should store either the base path or the full path with catalog name, but not mix the two approaches.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -73,10 +74,14 @@ public class Warehouse {
private static final String CAT_DB_TABLE_SEPARATOR = ".";
private Path whRoot;
+ private Path whCatRoot;
private Path whRootExternal;
+ private Path whCatRootExternal;
Review Comment:
The fields `whRoot`, `whCatRoot`, `whRootExternal`, and `whCatRootExternal` are not thread-safe. Multiple threads could simultaneously check if these fields are null and attempt to initialize them, leading to race conditions. These fields should be marked as `volatile` or the getter methods should be synchronized to ensure thread-safe lazy initialization.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
+ ICEBERG
+ }
+
+ public static final Set<CatalogType> VALID_CATALOG_TYPES = EnumSet.of(CatalogType.NATIVE, CatalogType.ICEBERG);
+
+ public static boolean isValidCatalogType(String name) {
+ CatalogType type = EnumUtils.getEnumIgnoreCase(CatalogType.class, name);
+ return type != null && VALID_CATALOG_TYPES.contains(type);
Review Comment:
The constant `VALID_CATALOG_TYPES` on line 32 is redundant because it contains exactly the same values as the `CatalogType` enum. The `isValidCatalogType` method could simply check if `type != null` after parsing with `EnumUtils.getEnumIgnoreCase`, making this constant unnecessary. This reduces maintainability as future catalog types need to be added in two places.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -156,33 +170,67 @@ public Path getDnsPath(Path path) throws MetaException {
* This involves opening the FileSystem corresponding to the warehouse root
* dir (but that should be ok given that this is only called during DDL
* statements for non-external tables).
+ *
+ * @deprecated use {@link #getWhRoot(String)}
*/
public Path getWhRoot() throws MetaException {
- if (whRoot != null) {
- return whRoot;
+ return getWhRoot(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRoot(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName.trim());
+
+ Path rootDir = isDefault ? whRoot : whCatRoot;
+ if (rootDir == null) {
+ rootDir = getDnsPath(new Path(isDefault ? whRootString : whCatRootString));
+ if (isDefault) {
+ whRoot = rootDir;
+ } else {
+ whCatRoot = rootDir;
+ }
}
- whRoot = getDnsPath(new Path(whRootString));
- return whRoot;
+
+ return isDefault ? rootDir : new Path(rootDir, catalogName);
}
+
+
+ /**
+ * @deprecated use {@link #getWhRootExternal(String)}
+ */
public Path getWhRootExternal() throws MetaException {
- if (whRootExternal != null) {
- return whRootExternal;
+ return getWhRootExternal(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRootExternal(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName);
+
+ Path rootExDir = isDefault ? whRootExternal : whCatRootExternal;
+ if (rootExDir != null) {
+ return isDefault ? rootExDir : new Path(rootExDir, catalogName);
}
- if (!hasExternalWarehouseRoot()) {
- whRootExternal = getWhRoot();
- } else {
- whRootExternal = getDnsPath(new Path(whRootExternalString));
+
+ if (isDefault) {
+ rootExDir = hasExternalWarehouseRoot()
+ ? getDnsPath(new Path(whRootExternalString))
+ : getWhRoot(catalogName);
+ whRootExternal = rootExDir;
+ return rootExDir;
}
- return whRootExternal;
+
+ rootExDir = hasExternalWarehouseRoot(catalogName)
+ ? getDnsPath(new Path(whCatRootExternalString, catalogName))
+ : getWhRoot(catalogName);
+ whCatRootExternal = rootExDir;
+ return rootExDir;
}
Review Comment:
The new methods `getWhRoot(String catalogName)` and `getWhRootExternal(String catalogName)` lack proper JavaDoc documentation. They should have JavaDoc comments explaining their purpose, parameters, return values, and exceptions. This is especially important since the old no-arg versions are now deprecated in favor of these.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java:
##########
@@ -81,34 +81,62 @@ public int execute() throws HiveException {
}
private void makeLocationQualified(Database database) throws HiveException {
+ String catalogName = database.getCatalogName().toLowerCase();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equalsIgnoreCase(catalogName);
+
+ // -------- External location --------
if (database.isSetLocationUri()) {
database.setLocationUri(Utilities.getQualifiedPath(context.getConf(), new Path(database.getLocationUri())));
} else {
- // Location is not set we utilize WAREHOUSE_EXTERNAL together with database name
- String rootDir = MetastoreConf.getVar(context.getConf(), MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL);
- if (rootDir == null || rootDir.trim().isEmpty()) {
- // Fallback plan
- LOG.warn(String.format(
- "%s is not set, falling back to %s. This could cause external tables to use to managed tablespace.",
- MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL.getVarname(), MetastoreConf.ConfVars.WAREHOUSE.getVarname()));
- rootDir = MetastoreConf.getVar(context.getConf(), MetastoreConf.ConfVars.WAREHOUSE);
- }
- Path path = new Path(rootDir, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
- String qualifiedPath = Utilities.getQualifiedPath(context.getConf(), path);
- database.setLocationUri(qualifiedPath);
+ String rootDir = getExternalRootDir(isDefaultCatalog);
+ Path path = buildDbPath(rootDir, catalogName, dbName, isDefaultCatalog);
+ database.setLocationUri(Utilities.getQualifiedPath(context.getConf(), path));
}
+ // -------- Managed location --------
if (database.isSetManagedLocationUri()) {
database.setManagedLocationUri(Utilities.getQualifiedPath(context.getConf(),
new Path(database.getManagedLocationUri())));
} else {
- // ManagedLocation is not set we utilize WAREHOUSE together with database name
- String rootDir = MetastoreConf.getVar(context.getConf(), MetastoreConf.ConfVars.WAREHOUSE);
- Path path = new Path(rootDir, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ String rootDir = MetastoreConf.getVar(
+ context.getConf(),
+ isDefaultCatalog
+ ? MetastoreConf.ConfVars.WAREHOUSE
+ : MetastoreConf.ConfVars.WAREHOUSE_CATALOG
+ );
+
+ Path path = buildDbPath(rootDir, catalogName, dbName, isDefaultCatalog);
String qualifiedPath = Utilities.getQualifiedPath(context.getConf(), path);
if (!qualifiedPath.equals(database.getLocationUri())) {
database.setManagedLocationUri(qualifiedPath);
}
}
}
+
+ private Path buildDbPath(String rootDir, String catalogName, String dbName, boolean isDefaultCatalog) {
+ return isDefaultCatalog
+ ? new Path(rootDir, dbName + DATABASE_PATH_SUFFIX)
+ : new Path(rootDir + "/" + catalogName, dbName + DATABASE_PATH_SUFFIX);
Review Comment:
The path concatenation on line 120 uses string concatenation (`rootDir + "/" + catalogName`) instead of using the Path constructor. This is problematic because it doesn't account for cases where `rootDir` might already end with a separator, potentially creating paths with double slashes. Use `new Path(new Path(rootDir), catalogName)` or similar Path-based construction for proper path handling.
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:
##########
@@ -388,9 +389,15 @@ public static void createDbExportDump(FileSystem fs, Path metadataPath, Database
private static void updateIfCustomDbLocations(Database database, Configuration conf) throws SemanticException {
try {
- String whLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL,
- MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE));
- Path dbDerivedLoc = new Path(whLocatoion, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equals(database.getCatalogName());
+ String whLocation = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL : MetastoreConf.ConfVars.WAREHOUSE_CATALOG_EXTERNAL,
+ MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE : MetastoreConf.ConfVars.WAREHOUSE_CATALOG
+ )
+ );
+
+ Path dbDerivedLoc = new Path(whLocation, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
Review Comment:
The path construction on line 400 doesn't account for non-default catalogs. When determining if a database has a custom location, the code builds `dbDerivedLoc` from `whLocation` and database name, but for non-default catalogs, the catalog name should also be included in the path (similar to how it's done in `CreateDatabaseOperation.buildDbPath()`). This could lead to incorrect detection of custom database locations for non-default catalogs.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1783,6 +1783,15 @@ public enum ConfVars {
"hive.metastore.warehouse.external.dir", "",
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ WAREHOUSE_CATALOG("metastore.warehouse.catalog.dir",
+ "hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat."),
+ WAREHOUSE_CATALOG_EXTERNAL("metastore.warehouse.catalog.external.dir",
+ "hive.metastore.warehouse.catalog.external.dir", "",
+ "Default location for external tables created in the warehouse. " +
+ "If not set or null, then the normal warehouse location(MetastoreConf.WAREHOUSE_CATALOG) " +
+ "will be used as the default location."),
Review Comment:
Missing space before the opening parenthesis in the description, similar to the issue in HiveConf. The text should read "...warehouse location (MetastoreConf.WAREHOUSE_CATALOG)" instead of "...warehouse location(MetastoreConf.WAREHOUSE_CATALOG)".
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -156,33 +170,67 @@ public Path getDnsPath(Path path) throws MetaException {
* This involves opening the FileSystem corresponding to the warehouse root
* dir (but that should be ok given that this is only called during DDL
* statements for non-external tables).
+ *
+ * @deprecated use {@link #getWhRoot(String)}
*/
public Path getWhRoot() throws MetaException {
- if (whRoot != null) {
- return whRoot;
+ return getWhRoot(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRoot(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName.trim());
+
+ Path rootDir = isDefault ? whRoot : whCatRoot;
+ if (rootDir == null) {
+ rootDir = getDnsPath(new Path(isDefault ? whRootString : whCatRootString));
+ if (isDefault) {
+ whRoot = rootDir;
+ } else {
+ whCatRoot = rootDir;
+ }
}
- whRoot = getDnsPath(new Path(whRootString));
- return whRoot;
+
+ return isDefault ? rootDir : new Path(rootDir, catalogName);
}
+
+
+ /**
+ * @deprecated use {@link #getWhRootExternal(String)}
+ */
public Path getWhRootExternal() throws MetaException {
- if (whRootExternal != null) {
- return whRootExternal;
+ return getWhRootExternal(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRootExternal(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName);
Review Comment:
There's an inconsistency in how catalog names are compared. In `getWhRoot()` on line 181, the code uses `catalogName.trim()` to handle whitespace, but in `getWhRootExternal()` on line 206, it compares directly without trimming. This inconsistency could lead to different behavior when catalog names have leading or trailing whitespace. Both methods should use the same comparison approach.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java:
##########
@@ -51,10 +54,16 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
@Override
public void analyzeInternal(ASTNode root) throws SemanticException {
Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));
- String catalogName = catDbNamePair.getLeft();
- if (catalogName != null && getCatalog(catalogName) == null) {
+ String catalogName = Optional.ofNullable(catDbNamePair.getLeft())
+ .orElse(HiveUtils.getCurrentCatalogOrDefault(conf));
+
+ Catalog catalog = getCatalog(catalogName);
+ if (catalog == null) {
throw new SemanticException(ErrorMsg.CATALOG_NOT_EXISTS, catalogName);
}
+ if (!DDLUtils.doesCatalogSupportCreateDB(catalog)) {
+ throw new SemanticException("Catalog %s does not support creating database".formatted(catalogName));
Review Comment:
The error message uses String.formatted() which was introduced in Java 15, but Hive typically supports earlier Java versions. This could cause compatibility issues. Consider using String.format() instead for broader Java version compatibility.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogOperation.java:
##########
@@ -37,7 +38,11 @@ public CreateCatalogOperation(DDLOperationContext context, CreateCatalogDesc des
@Override
public int execute() throws Exception {
- Catalog catalog = new Catalog(desc.getName(), desc.getLocationUri());
+ String catLocationUri = Optional.ofNullable(desc.getLocationUri())
+ .orElse(MetastoreConf.getVar(context.getConf(),
+ MetastoreConf.ConfVars.WAREHOUSE_CATALOG) + "/" + desc.getName());
Review Comment:
The path construction on line 43 uses string concatenation (`+ "/" +`) instead of proper Path construction. This is the same issue as in CreateDatabaseOperation - it doesn't handle cases where the warehouse catalog directory might already end with a separator, potentially creating paths with double slashes. Use `new Path(...)` constructor for proper path handling.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat"),
Review Comment:
The description for `METASTORE_CATALOG_WAREHOUSE` on lines 926-927 has the same missing space after the comma issue as in MetastoreConf. It should read "...is testcat, then the default..." instead of "...is testcat,then the default..."
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -156,33 +170,67 @@ public Path getDnsPath(Path path) throws MetaException {
* This involves opening the FileSystem corresponding to the warehouse root
* dir (but that should be ok given that this is only called during DDL
* statements for non-external tables).
+ *
+ * @deprecated use {@link #getWhRoot(String)}
*/
public Path getWhRoot() throws MetaException {
- if (whRoot != null) {
- return whRoot;
+ return getWhRoot(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRoot(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName.trim());
+
+ Path rootDir = isDefault ? whRoot : whCatRoot;
+ if (rootDir == null) {
+ rootDir = getDnsPath(new Path(isDefault ? whRootString : whCatRootString));
+ if (isDefault) {
+ whRoot = rootDir;
+ } else {
+ whCatRoot = rootDir;
+ }
}
- whRoot = getDnsPath(new Path(whRootString));
- return whRoot;
+
+ return isDefault ? rootDir : new Path(rootDir, catalogName);
}
+
+
+ /**
+ * @deprecated use {@link #getWhRootExternal(String)}
+ */
public Path getWhRootExternal() throws MetaException {
- if (whRootExternal != null) {
- return whRootExternal;
+ return getWhRootExternal(DEFAULT_CATALOG_NAME);
+ }
+
+ public Path getWhRootExternal(String catalogName) throws MetaException {
+ boolean isDefault = DEFAULT_CATALOG_NAME.equals(catalogName);
+
+ Path rootExDir = isDefault ? whRootExternal : whCatRootExternal;
+ if (rootExDir != null) {
+ return isDefault ? rootExDir : new Path(rootExDir, catalogName);
}
- if (!hasExternalWarehouseRoot()) {
- whRootExternal = getWhRoot();
- } else {
- whRootExternal = getDnsPath(new Path(whRootExternalString));
+
+ if (isDefault) {
+ rootExDir = hasExternalWarehouseRoot()
+ ? getDnsPath(new Path(whRootExternalString))
+ : getWhRoot(catalogName);
+ whRootExternal = rootExDir;
+ return rootExDir;
}
- return whRootExternal;
+
+ rootExDir = hasExternalWarehouseRoot(catalogName)
+ ? getDnsPath(new Path(whCatRootExternalString, catalogName))
+ : getWhRoot(catalogName);
+ whCatRootExternal = rootExDir;
+ return rootExDir;
Review Comment:
The same caching issue exists in `getWhRootExternal()`. When the cached value exists and is non-null (line 209), the method returns `new Path(rootExDir, catalogName)` for non-default catalogs. However, the cached value `whCatRootExternal` is set on line 224 to already include the catalog name (from line 222: `new Path(whCatRootExternalString, catalogName)`). This means subsequent calls will incorrectly append the catalog name again.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLUtils.java:
##########
@@ -292,4 +295,11 @@ public static Pair<String, String> getCatDbNamePair(ASTNode dbNameNode) throws S
return Pair.of(catName, dbName);
}
+
+ public static boolean doesCatalogSupportCreateDB(Catalog catalog) {
+ if (catalog.getName().equals(Warehouse.DEFAULT_CATALOG_NAME)) {
+ return true;
+ }
+ return CatalogUtil.CatalogType.NATIVE.toString().toLowerCase().equals(catalog.getParameters().get("type"));
Review Comment:
The method `doesCatalogSupportCreateDB` could throw a NullPointerException if `catalog.getParameters()` returns null or if the catalog type is stored with different casing. The code should handle the case where parameters might be null and should perform a case-insensitive comparison since catalog type validation in `CatalogUtil.isValidCatalogType` uses `EnumUtils.getEnumIgnoreCase`. Add null-safety checks for `catalog.getParameters()` and use case-insensitive comparison.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +326,59 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
+ Database db = new Database();
+ db.setName(dbName);
+ db.setCatalogName(DEFAULT_CATALOG_NAME);
+ return getDefaultDatabasePath(db, inExternalWH);
+ }
+
+ public Path getDefaultDatabasePath(Database db, boolean inExternalWH) throws MetaException {
+ String catalogName = db.getCatalogName();
+ String dbName = db.getName();
if (inExternalWH) {
if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
+ return getWhRootExternal(catalogName);
}
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ return new Path(getWhRootExternal(catalogName), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
} else {
if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
+ return getWhRoot(catalogName);
}
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ return new Path(getWhRoot(catalogName), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
}
}
Review Comment:
The new method `getDefaultDatabasePath(Database db, boolean inExternalWH)` lacks proper JavaDoc documentation. It should have JavaDoc comments explaining its purpose, parameters, return values, and exceptions, following the pattern of other public methods in this class.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +326,59 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
Review Comment:
The new methods `getDefaultDatabasePath(Database db)` and `getDefaultExternalDatabasePath(Database db)` lack proper JavaDoc documentation. They should have JavaDoc comments explaining their purpose, parameters, return values, and exceptions, following the pattern of other public methods in this class.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1783,6 +1783,15 @@ public enum ConfVars {
"hive.metastore.warehouse.external.dir", "",
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ WAREHOUSE_CATALOG("metastore.warehouse.catalog.dir",
+ "hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat."),
Review Comment:
The description for `WAREHOUSE_CATALOG` on lines 1788-1789 has a missing space after the comma. It should read "...is testcat, then the default..." instead of "...is testcat,then the default..."
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractRepository.java:
##########
@@ -82,7 +82,7 @@ protected Database initDatabase(Hive hive) {
db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
if (db == null) {
LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
- String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(db);
Review Comment:
The variable `db` is being used before it's initialized. On line 82, `db` is retrieved from `hive.getDatabase()`, but on line 85, if `db` is null, the code calls `getDatabaseLocation(db)` passing the null reference. This will cause a NullPointerException when the method tries to call `db.getCatalogName()` inside `getDefaultExternalDatabasePath(db)`. The database object should be created before calling `getDatabaseLocation()`.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
+ ICEBERG
+ }
+
+ public static final Set<CatalogType> VALID_CATALOG_TYPES = EnumSet.of(CatalogType.NATIVE, CatalogType.ICEBERG);
+
+ public static boolean isValidCatalogType(String name) {
+ CatalogType type = EnumUtils.getEnumIgnoreCase(CatalogType.class, name);
+ return type != null && VALID_CATALOG_TYPES.contains(type);
+ }
+}
Review Comment:
The class `CatalogUtil` and its public members lack JavaDoc documentation. Public classes and methods should have JavaDoc comments explaining their purpose, especially since this is a new API being introduced. The `CatalogType` enum should also document what each type represents and when to use it.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat"),
+
+ HIVE_METASTORE_CATALOG_WAREHOUSE_EXTERNAL("hive.metastore.warehouse.catalog.external.dir", null,
+ "Default location for external tables created in the warehouse. " +
+ "If not set or null, then the normal warehouse location(MetastoreConf.METASTORE_CATALOG_WAREHOUSE)" +
+ "will be used as the default location."),
Review Comment:
Missing space before the opening parenthesis. The text should read "...warehouse location (MetastoreConf.METASTORE_CATALOG_WAREHOUSE)" instead of "...warehouse location(MetastoreConf.METASTORE_CATALOG_WAREHOUSE)".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3754040889
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [76 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3760427927
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [63 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3010714929
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
> CREATE EXTERNAL CATALOG - Foreign catalog
> CREATE CATALOG - Native catalog
Sounds good. But I think the property "EXTERNAL" = {"true", "false"} is unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +--------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975527660
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
> @zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
Basically, it's about the same. If I add a Hudi catalog, I think it should be similar to the Iceberg catalog.
> isn't it enough to have just 2 categories
If I only use two catalog types, namely native and external, I'm not sure what the syntax would look like. The syntax I listed is mainly based on StarRocks and Doris, and I think Trino should be similar as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4111237882
> > 6. Every thing a catalog needs is stored in CATALOG_PARAMS, including the driver class, file system configurations, or extra jar files. The PARAM_VALUE should be a clob instead of varchar in CATALOG_PARAMS.
>
> what about StorageCredentialParams? would those be part of CATALOG_PARAMS, like location to secrets??? Not all catalogs support credential vending ATM. Or it would be part of client configs?
I think StorageCredentialParams would be part of CATALOG_PARAMS. For some Cloud service providers, such as AWS, StorageCredentialParams maybe a requirement, and StorageCredentialParams should be configurations passed from the client to the server, but for some other storage system like hdfs, no need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
"type" = {"hive", "iceberg", "jdbc", "glue", "rest", etc}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4229539540
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [69 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [1.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068050601
##########
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:
##########
@@ -132,6 +132,8 @@ public void authorizeDbLevelOperations(Privilege[] readRequiredPriv, Privilege[]
Path root = null;
try {
initWh();
+ // TODO catalog. Need to determine auth root path based on catalog name.
Review Comment:
could you please add the ticket to the EPIC
cc @saihemanth-cloudera, as he is the most experienced in this area
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3797696523
One thing to note: Currently, we have two entry points for creating catalogs—one is the SQL-based catalog creation entry involved in this PR, and the other is the HMS API-based catalog creation entry.
In this PR, I intend to enforce the inclusion of the `type` attribute when users create catalogs via SQL, so that users can clearly distinguish between different types of catalogs. However, the HMS API side does not enforce the `type` attribute when creating catalogs, which would lead to inconsistency between the two entry points.
Therefore, I have two approaches to resolve this issue and ensure consistency between the two entry points:
1) On the HMS server side, specifically in the `HMSHandler`, enforce a check during catalog creation to verify whether the user has provided the `type` attribute. If the `type` is missing, return an error. This approach would introduce backward compatibility issues for catalog creation.
2) Modify the current PR to make the `type` attribute optional for SQL-based catalog creation. For both SQL-based catalog creation and HMS API-based catalog creation, if the user does not provide the `type` attribute, assign a default value (e.g., `native`) on the HMSHandler server side. This approach would not cause compatibility issues with the catalog creation syntax. However, other open-source components enforce the `type` attribute requirement, and the `type` helps users better understand the parameter requirements for the catalog, among other benefits.
What are your suggestions regarding these two approaches? Thanks in advance.
cc @deniskuzZ @difin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068014651
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
We had some discussion in the document: https://docs.google.com/document/d/1SX8OPd_KdBuynMr-D8AZ7V5tKkoUDKFterUSjeRpOW0/edit?disco=AAAB2orynk4
Wechat gave a good discussion:
> Also, since both native and hive point to Hive tables, it might be cleaner to unify them as type = hive, and use another field to distinguish local HMS vs external HMS.
>
> In this way it might be clearer to separate “catalog type” from “catalog scope / connectivity”, and type consistently represents the data source now.
So I’m considering: for the internal HMS catalog, should we use the "native" type or the "hive" type? If we use the "hive" type for native hms catalog, then the main difference between a remote/external Hive catalog and a native/internal Hive catalog would be whether the `external` keyword is present when creating the catalog.
@deniskuzZ What do you think — should we unify remote and internal Hive catalogs to both use the "hive" type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068014153
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
Yeah, that’s exactly what I was trying to suggest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137315736
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +330,56 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
- if (inExternalWH) {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
- }
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
- } else {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
- }
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ Database db = new Database();
Review Comment:
CatalogDb is a self-describing, single-purpose DTO. It clearly communicates intent, keeps the data model focused, and avoids overloading more generic structures.
Note that was only a suggestion, and if you don’t like it, that’s totally fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4294922010
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [73 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [1.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809561947
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
NATIVE means Hive only?
if we plan support federation won't REMOTE be a better choice instead of ICEBERG? Iceberg is a table type
@dengzhhu653, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3134769362
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -694,6 +694,9 @@ private void rollbackTransactionToSavePoint(String savePoint) {
@Override
public void createCatalog(Catalog cat) throws MetaException {
LOG.debug("Creating catalog {}", cat);
+
+ ensureCatalogType(cat);
Review Comment:
For user input check, it's better to perform at the hander, ObjectStore is where we manage the metadata.
For negative impacts, we need to prepare an ObjectStore though it's invalid and might take a connection from the connection pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4229454001
> > One thing to note: Currently, we have two entry points for creating catalogs—one is the SQL-based catalog creation entry involved in this PR, and the other is the HMS API-based catalog creation entry. In this PR, I intend to enforce the inclusion of the `type` attribute when users create catalogs via SQL, so that users can clearly distinguish between different types of catalogs. However, the HMS API side does not enforce the `type` attribute when creating catalogs, which would lead to inconsistency between the two entry points.
> > Therefore, I have two approaches to resolve this issue and ensure consistency between the two entry points:
> >
> > 1. On the HMS server side, specifically in the `HMSHandler`, enforce a check during catalog creation to verify whether the user has provided the `type` attribute. If the `type` is missing, return an error. This approach would introduce backward compatibility issues for catalog creation.
> > 2. Modify the current PR to make the `type` attribute optional for SQL-based catalog creation. For both SQL-based catalog creation and HMS API-based catalog creation, if the user does not provide the `type` attribute, assign a default value (e.g., `native`) on the HMSHandler server side. This approach would not cause compatibility issues with the catalog creation syntax. However, other open-source components enforce the `type` attribute requirement, and the `type` helps users better understand the parameter requirements for the catalog, among other benefits.
> >
> > What are your suggestions regarding these two approaches? Thanks in advance. cc @deniskuzZ @difin
>
> @zhangbutao i would vote for option 2, fallback to default - `native`
Changes have been made to keep the type optional when creating a catalog. Thanks for the advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068014651
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
We had some discussion in the document: https://docs.google.com/document/d/1SX8OPd_KdBuynMr-D8AZ7V5tKkoUDKFterUSjeRpOW0/edit?disco=AAAB2orynk4
Wechar @wecharyu gave a good discussion:
> Also, since both native and hive point to Hive tables, it might be cleaner to unify them as type = hive, and use another field to distinguish local HMS vs external HMS.
>
> In this way it might be clearer to separate “catalog type” from “catalog scope / connectivity”, and type consistently represents the data source now.
So I’m considering: for the internal HMS catalog, should we use the "native" type or the "hive" type? If we use the "hive" type for native hms catalog, then the main difference between a remote/external Hive catalog and a native/internal Hive catalog would be whether the `external` keyword is present when creating the catalog.
@deniskuzZ What do you think — should we unify remote and internal Hive catalogs to both use the "hive" type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068031057
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
+ }
+ checkCatalogType(props);
+
CreateCatalogDesc desc = new CreateCatalogDesc(catalogName, comment, locationUrl, ifNotExists, props);
Catalog catalog = new Catalog(catalogName, locationUrl);
rootTasks.add(TaskFactory.get(new DDLWork(getInputs(), getOutputs(), desc)));
outputs.add(new WriteEntity(catalog, WriteEntity.WriteType.DDL_NO_LOCK));
}
+
+ private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+ String catalogType = props.get("type");
+ // Set default type to native if not specified
+ if (Strings.isNullOrEmpty(catalogType)) {
+ props.put("type", CatalogUtil.NATIVE);
+ return;
+ }
+ // Normalize and validate catalog type (fail fast on invalid input)
+ try {
+ props.put("type", CatalogUtil.normalizeCatalogType(catalogType));
Review Comment:
Could we simplify by handling isNullOrEmpty in `normalizeCatalogType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068037393
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java:
##########
@@ -81,34 +81,62 @@ public int execute() throws HiveException {
}
private void makeLocationQualified(Database database) throws HiveException {
+ String catalogName = database.getCatalogName().toLowerCase();
Review Comment:
Shouldn't the names be normalized here: special chars/encoding lowercase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068059463
##########
ql/src/test/queries/clientpositive/catalog.q:
##########
@@ -7,22 +7,22 @@ dfs -mkdir -p hdfs:///tmp/test_cat;
-- SORT_QUERY_RESULTS
SHOW CATALOGS;
--- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');
Review Comment:
same `'type'='NATIVE'` is redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. what confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under the catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3825740815
hi @zhangbutao, thanks for the PR! I’ll try to get back with my thoughts on the topic next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | ---------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4794928278
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4089024948
Some ideas:
1, We have two catalog types, one is Native, the other is Non-Native;
2. For Native catalog, besides get table, the client is able to create/drop/alter tables/partitions, so the catalog location uri is a must, we can check it upon creating the catalog;
3. For Non-Native catalog, it's just a "mount" of external metadata sources, no changes allowed;
4. Native can be changed to a Non-Native catalog, the Non-Native can be to a Native catalog as well;
5. For default native hive catalog, the catalog location is `hive.metastore.warehouse.dir`
6. Every thing a catalog needs is stored in CATALOG_PARAMS, including the driver class, file system configurations, or extra jar files. The PARAM_VALUE should be a clob instead of varchar in CATALOG_PARAMS.
Given these above, we don't need extra `metastore.warehouse.catalog.dir` or others to determine the default catalog location, the location could be anywhere, just make sure it's accessible by the Metastore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3762944300
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [63 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +---------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | | External HMS |
| Hive tables | | Hive tables |
| Iceberg | | Iceberg tables |
| tables | +-----------------+
| | | REST Catalog |
| | | Iceberg tables |
+-------------+ +-----------------+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "EXTERNAL" = {"true" , "false"}
'type' = {'hive', 'iceberg', 'jdbc', etc}
'hive.metastore.type' = {"hive', 'glue'}
'iceberg.catalog.type' = {'hive', 'glue', 'rest'}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
"type" = {"hive", "iceberg", "jdbc", etc}
"hive.metastore.type" = {"hive", "glue"}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994570029
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
cc @difin
what configs are you using to configure the iceberg rest catalog client in Hive ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4124754013
> tables and databases under the newly created native catalog do not respect its own catalog location.
Let's address it separately, check the catalog's location when creating the database.
I don't like assuming we must have a location in Catalog, for non-native catalog(Let's rename it to Foreign catalog), such as JDBC, Kudu, Kafka(the catalog type), this location is unnecessary IMO.
It's also doesn't mean every native catalog should be reside under the same parent directory or file system. For example, in a hybrid env the catalog directory could be a S3 bucket or a hdfs directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
Sorry, I might be missing something. Where is the confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)? Could you please elaborate?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
I'm aligned on this as well
+1
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068071794
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +330,56 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
- if (inExternalWH) {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
- }
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
- } else {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
- }
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ Database db = new Database();
Review Comment:
are you using Database only as a DTO object? maybe create a record CatalogDb as mentioned earlier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068066820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##########
@@ -631,6 +631,8 @@ private static void setCmRootPermissions(Path cmroot) throws IOException{
FileSystem cmFs = cmroot.getFileSystem(conf);
cmFs.setPermission(cmroot, new FsPermission("770"));
try {
+ // TODO catalog. The Repl function is currently only available for the default catalog path ConfVars.WAREHOUSE.
Review Comment:
should we add the ticket to the EPIC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068060902
##########
ql/src/test/queries/clientpositive/catalog.q:
##########
@@ -7,22 +7,22 @@ dfs -mkdir -p hdfs:///tmp/test_cat;
-- SORT_QUERY_RESULTS
SHOW CATALOGS;
--- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');
-- DESCRIBE
DESC CATALOG test_cat;
-- CREATE INE already exists
-CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat';
+CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat' PROPERTIES('type'='native');
Review Comment:
do we have tests for external catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068142421
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
sure, why not, since we'll have an external flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3134741096
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import com.google.common.base.Strings;
+
+public class CatalogUtil {
+ public static final String TYPE = "type";
+
+ // Supported Catalog types
+ public enum CatalogType {
+ HIVE,
Review Comment:
what's the purpose of `CatalogType`? if it's used to describe the source, `CatalogConnector` or `CatalogScope` would be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3083881872
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import com.google.common.base.Strings;
+
+public class CatalogUtil {
+ public static final String TYPE = "type";
+
+ // Supported Catalog types
+ public enum CatalogType {
+ HIVE,
Review Comment:
Is this `CatalogType` for what not where? if it's for what, maybe named as `CatalogConnector` or something else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974723922
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
StarRock also uses [hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/) and [iceberg](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog/) types in `CREATE EXTERNAL CATALOG` cmd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type. I assume for GLUE, there would be "glue" type, right?
- native,
- jdbc,
- iceberg (hive/rest),
- hive,
- glue
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | ---------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2979582881
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
Good catch. I think I’ve been mixing up the use of connector type and catalog type. However, it seems that different open-source components define connectors and catalogs differently. For example, Trino uses the term connector to describe multi-source federated queries, as seen in https://trino.io/docs/current/connector.html. On the other hand, StarRocks and Apache Doris use the catalog to describe federated query capabilities.
In Apache Gravitino, the usage of the catalogs and connector has a different positioning. In Gravitino, catalogs refer to the various data sources that the Gravitino server can uniformly manage, as shown in https://github.com/apache/gravitino/tree/main/catalogs. Meanwhile, connectors refer to the various engines that implement their own read/write plugins to read and write multiple types of data through Gravitino, such as in https://github.com/apache/gravitino/tree/main/spark-connector.
So, in Hive, how should we define the use of connector and catalog? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3023613734
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
might be. i was just thinking that when you describe catalog - we need to show the flag "EXTERNAL" = {"true", "false"} or similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137409235
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogOperation.java:
##########
@@ -37,7 +38,11 @@ public CreateCatalogOperation(DDLOperationContext context, CreateCatalogDesc des
@Override
public int execute() throws Exception {
- Catalog catalog = new Catalog(desc.getName(), desc.getLocationUri());
+ String catLocationUri = Optional.ofNullable(desc.getLocationUri())
Review Comment:
we can simplify naming to just `locationUri` since class is about Catalog anyways
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068032430
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogOperation.java:
##########
@@ -37,7 +38,11 @@ public CreateCatalogOperation(DDLOperationContext context, CreateCatalogDesc des
@Override
public int execute() throws Exception {
- Catalog catalog = new Catalog(desc.getName(), desc.getLocationUri());
+ String catLocationUri = Optional.ofNullable(desc.getLocationUri())
Review Comment:
minor. i think `cat` prefix in local var is redundant here since we are inside CreateCatalog class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068038313
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java:
##########
@@ -81,34 +81,62 @@ public int execute() throws HiveException {
}
private void makeLocationQualified(Database database) throws HiveException {
+ String catalogName = database.getCatalogName().toLowerCase();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equalsIgnoreCase(catalogName);
Review Comment:
consider moving to utils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068056716
##########
ql/src/test/queries/clientnegative/lockneg_try_lock_cat_db_in_use.q:
##########
@@ -1,7 +1,7 @@
set hive.lock.numretries=0;
set hive.support.concurrency=true;
-CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog';
+CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog' PROPERTIES('type'='native');
Review Comment:
Could we make catalogs “native” by default unless a different option is specified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/table # non default catalog
/warehouse/<database_name>.db/table # default catalog
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/table # non default catalog
/warehouse/<database_name>.db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/catalog/db/table # non default catalog
/warehouse/db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3069111332
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
> shouldn't the catalog be under the warehousePath?
No. The `warehousePath` is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the `warehousePath`, it will cause confusion with the path information of Hive databases under the default catalog.
The path information for the default catalog's databases is controlled by the property `metastore.warehouse.dir`, which defaults to `/user/hive/warehouse`. The path for non-default catalogs is controlled by the new added property`metastore.warehouse.catalog.dir`, which defaults to `/user/hive/catalog/warehouse`.
```
/user/hive/catalog/warehouse/<catalog_name>/<database_name>.db/<table_name>/ # non default catalog
/user/hive/warehouse/<database_name>.db/<table_name>/ # default catalog
/user/hive/warehouse/default.db/<table_name>/ # default catalog & database
```
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogOperation.java:
##########
@@ -37,7 +38,11 @@ public CreateCatalogOperation(DDLOperationContext context, CreateCatalogDesc des
@Override
public int execute() throws Exception {
- Catalog catalog = new Catalog(desc.getName(), desc.getLocationUri());
+ String catLocationUri = Optional.ofNullable(desc.getLocationUri())
Review Comment:
Sorry, i didn't get you here. Could you please describe it in detail? Thank you.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java:
##########
@@ -81,34 +81,62 @@ public int execute() throws HiveException {
}
private void makeLocationQualified(Database database) throws HiveException {
+ String catalogName = database.getCatalogName().toLowerCase();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equalsIgnoreCase(catalogName);
Review Comment:
Fixed. Added a util method:
`HiveUtils.isDefaultCatalog(String catName, Configuration conf)`
##########
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:
##########
@@ -132,6 +132,8 @@ public void authorizeDbLevelOperations(Privilege[] readRequiredPriv, Privilege[]
Path root = null;
try {
initWh();
+ // TODO catalog. Need to determine auth root path based on catalog name.
Review Comment:
Create ticket HIVE-29562
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -86,17 +90,26 @@ public class Warehouse {
public Warehouse(Configuration conf) throws MetaException {
this.conf = conf;
- whRootString = MetastoreConf.getVar(conf, ConfVars.WAREHOUSE);
- if (StringUtils.isBlank(whRootString)) {
- throw new MetaException(ConfVars.WAREHOUSE.getVarname()
- + " is not set in the config or blank");
- }
+
+ whRootString = getRequiredVar(conf, ConfVars.WAREHOUSE);
+ whCatRootString = getRequiredVar(conf, ConfVars.WAREHOUSE_CATALOG);
Review Comment:
Already repied above.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java:
##########
@@ -51,10 +52,9 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
@Override
public void analyzeInternal(ASTNode root) throws SemanticException {
Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));
Review Comment:
I think using a `Pair `here is already simple and clear enough, there's no need to use the `record CatalogDb`.
##########
standalone-metastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java:
##########
@@ -79,6 +79,7 @@ private Catalog createCatalog() {
if (configUri != null && !configUri.isEmpty()) {
properties.put("uri", configUri);
}
+ // TODO catalog. Consider adding new created native catalog warehouse(MetastoreConf.ConfVars.WAREHOUSE_CATALOG) later.
Review Comment:
Fixed. This just maybe a question. I created a ticket HIVE-28879 for this.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
+ }
+ checkCatalogType(props);
+
CreateCatalogDesc desc = new CreateCatalogDesc(catalogName, comment, locationUrl, ifNotExists, props);
Catalog catalog = new Catalog(catalogName, locationUrl);
rootTasks.add(TaskFactory.get(new DDLWork(getInputs(), getOutputs(), desc)));
outputs.add(new WriteEntity(catalog, WriteEntity.WriteType.DDL_NO_LOCK));
}
+
+ private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+ String catalogType = props.get("type");
Review Comment:
Sure. Create constan `TYPE` in CatalogUtil
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:
##########
@@ -388,14 +389,34 @@ public static void createDbExportDump(FileSystem fs, Path metadataPath, Database
private static void updateIfCustomDbLocations(Database database, Configuration conf) throws SemanticException {
try {
- String whLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL,
- MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE));
- Path dbDerivedLoc = new Path(whLocatoion, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ String catName = database.getCatalogName();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equals(catName);
+
+ // external warehouse root
+ String whLocation = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL : MetastoreConf.ConfVars.WAREHOUSE_CATALOG_EXTERNAL,
+ MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE : MetastoreConf.ConfVars.WAREHOUSE_CATALOG));
+
+ if (!isDefaultCatalog) {
+ whLocation = new Path(whLocation, catName).toString();
+ }
+
+ Path dbDerivedLoc = new Path(whLocation, dbName + DATABASE_PATH_SUFFIX);
String defaultDbLoc = Utilities.getQualifiedPath((HiveConf) conf, dbDerivedLoc);
database.putToParameters(ReplConst.REPL_IS_CUSTOM_DB_LOC,
Boolean.toString(!defaultDbLoc.equals(database.getLocationUri())));
- String whManagedLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE);
- Path dbDerivedManagedLoc = new Path(whManagedLocatoion, database.getName().toLowerCase()
+
+ // managed warehouse root
+ String whManagedLocatoion = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE
Review Comment:
Replied in https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
Review Comment:
Sure. Changed.
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractRepository.java:
##########
@@ -82,7 +85,11 @@ protected Database initDatabase(Hive hive) {
db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
if (db == null) {
LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
- String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database();
Review Comment:
Fixed.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseOperation.java:
##########
@@ -81,34 +81,62 @@ public int execute() throws HiveException {
}
private void makeLocationQualified(Database database) throws HiveException {
+ String catalogName = database.getCatalogName().toLowerCase();
Review Comment:
At this stage, the database is being created, and the prerequisite is that a valid catalog already exists — there is no need to double validate the catalog here.
##########
ql/src/test/queries/clientnegative/lockneg_try_lock_cat_db_in_use.q:
##########
@@ -1,7 +1,7 @@
set hive.lock.numretries=0;
set hive.support.concurrency=true;
-CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog';
+CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog' PROPERTIES('type'='native');
Review Comment:
Removed.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
+ }
+ checkCatalogType(props);
+
CreateCatalogDesc desc = new CreateCatalogDesc(catalogName, comment, locationUrl, ifNotExists, props);
Catalog catalog = new Catalog(catalogName, locationUrl);
rootTasks.add(TaskFactory.get(new DDLWork(getInputs(), getOutputs(), desc)));
outputs.add(new WriteEntity(catalog, WriteEntity.WriteType.DDL_NO_LOCK));
}
+
+ private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+ String catalogType = props.get("type");
+ // Set default type to native if not specified
+ if (Strings.isNullOrEmpty(catalogType)) {
+ props.put("type", CatalogUtil.NATIVE);
+ return;
+ }
+ // Normalize and validate catalog type (fail fast on invalid input)
+ try {
+ props.put("type", CatalogUtil.normalizeCatalogType(catalogType));
Review Comment:
Sure. fixed.
##########
ql/src/test/queries/clientpositive/catalog.q:
##########
@@ -7,22 +7,22 @@ dfs -mkdir -p hdfs:///tmp/test_cat;
-- SORT_QUERY_RESULTS
SHOW CATALOGS;
--- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');
Review Comment:
Removed.
I also left a few DDL statements with 'type' = 'hive' to indicate that this parameter can be omitted for the Hive internal catalog.
##########
ql/src/test/queries/clientpositive/catalog.q:
##########
@@ -7,22 +7,22 @@ dfs -mkdir -p hdfs:///tmp/test_cat;
-- SORT_QUERY_RESULTS
SHOW CATALOGS;
--- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');
-- DESCRIBE
DESC CATALOG test_cat;
-- CREATE INE already exists
-CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat';
+CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat' PROPERTIES('type'='native');
Review Comment:
Currently, this PR is specifically targeting the internal/native catalog. If we were to implement a new external catalog, such as a JDBC catalog, I believe it would still require a lot of additional preliminary work(such as https://github.com/apache/hive/pull/6252), so we can consider doing it later.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+public class CatalogUtil {
+ // Catalog type constants
+ public static final String NATIVE = "native";
Review Comment:
Fixed.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##########
@@ -631,6 +631,8 @@ private static void setCmRootPermissions(Path cmroot) throws IOException{
FileSystem cmFs = cmroot.getFileSystem(conf);
cmFs.setPermission(cmroot, new FsPermission("770"));
try {
+ // TODO catalog. The Repl function is currently only available for the default catalog path ConfVars.WAREHOUSE.
Review Comment:
Sure. See HIVE-29278
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -279,27 +330,56 @@ public Path getDefaultDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, false);
}
+ public Path getDefaultDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, false);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultExternalDatabasePath(Database)}
+ */
public Path getDefaultExternalDatabasePath(String dbName) throws MetaException {
return getDefaultDatabasePath(dbName, true);
}
+ public Path getDefaultExternalDatabasePath(Database db) throws MetaException {
+ return getDefaultDatabasePath(db, true);
+ }
+
+ /**
+ * @deprecated use {@link #getDefaultDatabasePath(Database, boolean)}
+ */
// should only be used to determine paths before the creation of databases
public Path getDefaultDatabasePath(String dbName, boolean inExternalWH) throws MetaException {
- if (inExternalWH) {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRootExternal();
- }
- return new Path(getWhRootExternal(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
- } else {
- if (dbName.equalsIgnoreCase(DEFAULT_DATABASE_NAME)) {
- return getWhRoot();
- }
- return new Path(getWhRoot(), dbName.toLowerCase() + DATABASE_WAREHOUSE_SUFFIX);
+ Database db = new Database();
Review Comment:
I think this should work well. Regarding the changes made here, could you help me point out the more prominent advantages of using `record CatalogDb`?
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3083938782
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
I would like to raise my concern, personally I don't like to add a new `metastore.warehouse.catalog.dir` as a default location for new catalogs(non-default). this could bring some concerns IMO:
1. We use catalog to describe the source, we should at least get what/where the catalog is before creating the new catalog, that suggests don't create the catalog at will.
2. On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog, this might add some confusions on audit and bill consuming.
3. It's not a good experience to change the location after the catalog created, as we should consider moving the managed tables, which could be time/cost consuming.
So I would argue the location of catalog is immutable, we should check the location of non-default catalog on creating it, this is not a big change I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3757983983
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [59 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809561947
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
NATIVE means Hive only?
if we plan support federation won't REMOTE be a better choice instead of ICEBERG? Iceberg is a table type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2979582881
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
Good catch. I think I’ve been mixing up the use of connector type and catalog type. However, it seems that different open-source components define connectors and catalogs differently. For example, Trino uses the term connector to describe multi-source federated queries, as seen in https://trino.io/docs/current/connector.html. On the other hand, StarRocks and Apache Doris use the term catalog to describe federated query capabilities.
In Apache Gravitino, the usage of the terms catalog and connector has a different positioning. In Gravitino, catalogs refer to the various data sources that the Gravitino server can uniformly manage, as shown in https://github.com/apache/gravitino/tree/main/catalogs. Meanwhile, connectors refer to the various engines that implement their own read/write plugins to read and write multiple types of data through Gravitino, such as in https://github.com/apache/gravitino/tree/main/spark-connector.
So, in Hive, how should we define the use of connector and catalog? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4046031561
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [61 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4200256286
i ok with default catalog location for native catalogs if not explicitly provided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137321161
##########
ql/src/test/queries/clientpositive/catalog.q:
##########
@@ -7,22 +7,22 @@ dfs -mkdir -p hdfs:///tmp/test_cat;
-- SORT_QUERY_RESULTS
SHOW CATALOGS;
--- CREATE with comment
-CREATE CATALOG test_cat LOCATION 'hdfs:///tmp/test_cat' COMMENT 'Hive test catalog';
+-- CREATE with comment and default location
+CREATE CATALOG test_cat COMMENT 'Hive test catalog' PROPERTIES('type'='NATIVE');
-- DESCRIBE
DESC CATALOG test_cat;
-- CREATE INE already exists
-CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat';
+CREATE CATALOG IF NOT EXISTS test_cat LOCATION 'hdfs:///tmp/test_cat' PROPERTIES('type'='native');
Review Comment:
oh, ok. i didn't realized that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4229454783
> hi @zhangbutao, is the PR ready for 2nd round of review and includes the changes we discussed here?
Let's go :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068068477
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
##########
@@ -86,17 +90,26 @@ public class Warehouse {
public Warehouse(Configuration conf) throws MetaException {
this.conf = conf;
- whRootString = MetastoreConf.getVar(conf, ConfVars.WAREHOUSE);
- if (StringUtils.isBlank(whRootString)) {
- throw new MetaException(ConfVars.WAREHOUSE.getVarname()
- + " is not set in the config or blank");
- }
+
+ whRootString = getRequiredVar(conf, ConfVars.WAREHOUSE);
+ whCatRootString = getRequiredVar(conf, ConfVars.WAREHOUSE_CATALOG);
Review Comment:
please see the comment on default FS structure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068056716
##########
ql/src/test/queries/clientnegative/lockneg_try_lock_cat_db_in_use.q:
##########
@@ -1,7 +1,7 @@
set hive.lock.numretries=0;
set hive.support.concurrency=true;
-CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog';
+CREATE CATALOG testcat LOCATION '/tmp/testcat' COMMENT 'Hive test catalog' PROPERTIES('type'='native');
Review Comment:
Could we make catalogs “native” by default unless specified differently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive/default.db/<table_name>/ # default catalog & database
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
/warehouse/<catalog_name>.cat/<database_name>.db/table # non default catalog
/warehouse/<database_name>.db/table # default catalog
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4110308633
> Some ideas: 1, We have two catalog types, one is Native, the other is Non-Native; 2. For Native catalog, besides get table, the client can create/drop/alter tables/partitions, so the catalog location uri is a must, we can check it upon creating the catalog; 3. For Non-Native catalog, it's just a "mount" of external metadata sources, no changes allowed; 4. Native can be changed to a Non-Native catalog, the Non-Native can be to a Native catalog as well; 5. For default native hive catalog, the catalog location is `hive.metastore.warehouse.dir` 6. Every thing a catalog needs is stored in CATALOG_PARAMS, including the driver class, file system configurations, or extra jar files. The PARAM_VALUE should be a clob instead of varchar in CATALOG_PARAMS.
>
> Given these above, we don't need extra `metastore.warehouse.catalog.dir` or others to determine the default catalog location, the location could be anywhere, just make sure it's accessible by the Metastore.
1. Yes. Non-native catalogs may include JDBC catalog, another external HMS catalog, etc.
2. The catalog location can be set with a default value, so that users do not need to specify the catalog location every time they create a native catalog, which improves the user experience.
3. Non-native catalogs store data and metadata in external data sources, so they do not require a catalog location.
4. Changing a native catalog to a non-native one, or vice versa, seems unnecessary and infeasible. For example, a JDBC catalog cannot be converted into a native catalog.
5. Yes. The current default native catalog is named "hive", and its catalog location is determined by `hive.metastore.warehouse.dir`.
6. Good catch! `CATALOG_PARAMS` is used to store various properties of catalogs. However, not all information needs to be persisted in the HMS. For a large amount of configuration information, such as Hadoop-related configs like `hdfs-site.xml` and `core-site.xml` required by a non-native catalog, a local or cloud directory can be specified for the non-native catalog to dynamically load these configurations. Of course, if the `varchar` column's storage capacity becomes insufficient in the future, we will continue to optimize it.
> Given these above, we don't need extra `metastore.warehouse.catalog.dir` or others to determine the default catalog location. The location could be anywhere, just make sure it's accessible by the Metastore.
I’d like to emphasize the purpose of catalog location:
Catalog location is designed for native catalogs. Currently, after creating a new native catalog, even if the catalog's location is set differently from the default native catalog's location (`hive.metastore.warehouse.dir`), the created Hive databases still reside under `hive.metastore.warehouse.dir`. That is, tables and databases under the newly created native catalog do not respect its own catalog location. Therefore, this PR aims to address two issues:
1) Tables and databases under a newly created native catalog must respect its own catalog location and should not be mixed with the default native catalog (i.e., `hive.metastore.warehouse.dir`).
2) When creating a native catalog, the location attribute should be allowed to be omitted. The default value can be controlled by the parameter `metastore.warehouse.catalog.dir`, making it more convenient for users to create native catalogs, similar to how omitting the location when creating a Hive database results in a default database location.
@dengzhhu653 Please also refer to https://docs.google.com/document/d/1SX8OPd_KdBuynMr-D8AZ7V5tKkoUDKFterUSjeRpOW0/edit?tab=t.0 for more info.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4110393222
> 6. Every thing a catalog needs is stored in CATALOG_PARAMS, including the driver class, file system configurations, or extra jar files. The PARAM_VALUE should be a clob instead of varchar in CATALOG_PARAMS.
what about StorageCredentialParams? would those be part of CATALOG_PARAMS, like store location to secrets??? Not all catalogs support credential vending ATM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2700776326
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java:
##########
@@ -51,10 +52,9 @@ public CreateDatabaseAnalyzer(QueryState queryState) throws SemanticException {
@Override
public void analyzeInternal(ASTNode root) throws SemanticException {
Pair<String, String> catDbNamePair = DDLUtils.getCatDbNamePair((ASTNode) root.getChild(0));
- String catalogName = catDbNamePair.getLeft();
- if (catalogName != null && getCatalog(catalogName) == null) {
- throw new SemanticException(ErrorMsg.CATALOG_NOT_EXISTS, catalogName);
- }
+ String catalogName = Optional.ofNullable(catDbNamePair.getLeft())
+ .orElse(HiveUtils.getCurrentCatalogOrDefault(conf));
+
Review Comment:
The removed catalog existence validation (checking if getCatalog returns null) has been replaced with simply using the current catalog as default when catalog name is not specified. However, this removes an important validation step that would catch attempts to create databases in non-existent catalogs early in the analysis phase. Consider adding validation to verify that the catalog exists before proceeding with database creation, either here or in a later stage.
```suggestion
Object catalog = getCatalog(catalogName);
if (catalog == null) {
throw new SemanticException("Catalog " + catalogName + " does not exist");
}
```
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:
##########
@@ -388,14 +389,34 @@ public static void createDbExportDump(FileSystem fs, Path metadataPath, Database
private static void updateIfCustomDbLocations(Database database, Configuration conf) throws SemanticException {
try {
- String whLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL,
- MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE));
- Path dbDerivedLoc = new Path(whLocatoion, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ String catName = database.getCatalogName();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equals(catName);
+
+ // external warehouse root
+ String whLocation = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL : MetastoreConf.ConfVars.WAREHOUSE_CATALOG_EXTERNAL,
+ MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE : MetastoreConf.ConfVars.WAREHOUSE_CATALOG));
+
+ if (!isDefaultCatalog) {
+ whLocation += "/" + catName;
+ }
+
+ Path dbDerivedLoc = new Path(whLocation, dbName + DATABASE_PATH_SUFFIX);
String defaultDbLoc = Utilities.getQualifiedPath((HiveConf) conf, dbDerivedLoc);
database.putToParameters(ReplConst.REPL_IS_CUSTOM_DB_LOC,
Boolean.toString(!defaultDbLoc.equals(database.getLocationUri())));
- String whManagedLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE);
- Path dbDerivedManagedLoc = new Path(whManagedLocatoion, database.getName().toLowerCase()
+
+ // managed warehouse root
+ String whManagedLocatoion = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE
+ : MetastoreConf.ConfVars.WAREHOUSE_CATALOG);
+
+ if (!isDefaultCatalog) {
+ whManagedLocatoion += "/" + catName;
Review Comment:
String concatenation is used to build paths instead of using the Path API. Replace 'whManagedLocatoion += "/" + catName' with proper Path usage: 'whManagedLocatoion = new Path(whManagedLocatoion, catName).toString()' for consistency and correct path separator handling.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat"),
Review Comment:
The description reads 'location of default database for the warehouse' which is identical to the WAREHOUSE configuration description. This should be clarified to specify that this is the base location for databases in non-default catalogs. Consider rewording to: 'Base location for databases in non-default catalogs. The actual database path will be {this_value}/{catalog_name}/{database_name}.db'
```suggestion
"Base location for databases in non-default catalogs. " +
"The actual database path will be {this_value}/{catalog_name}/{database_name}.db"),
```
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1783,6 +1783,15 @@ public enum ConfVars {
"hive.metastore.warehouse.external.dir", "",
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ WAREHOUSE_CATALOG("metastore.warehouse.catalog.dir",
+ "hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat."),
Review Comment:
The documentation for WAREHOUSE_CATALOG is missing a space after the period in the description. The text 'If the name of the catalog to which the db belongs is testcat,then the default path' should be 'If the name of the catalog to which the db belongs is testcat, then the default path' (note the space after the comma).
```suggestion
" then the default path prefix for the db is /user/hive/catalog/warehouse/testcat."),
```
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +72,23 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ assert props != null;
+ checkCatalogType(props);
+
CreateCatalogDesc desc = new CreateCatalogDesc(catalogName, comment, locationUrl, ifNotExists, props);
Catalog catalog = new Catalog(catalogName, locationUrl);
rootTasks.add(TaskFactory.get(new DDLWork(getInputs(), getOutputs(), desc)));
outputs.add(new WriteEntity(catalog, WriteEntity.WriteType.DDL_NO_LOCK));
}
+
+ private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+ String catalogType = props.get("type");
+ if (Strings.isNullOrEmpty(catalogType)) {
+ throw new SemanticException("'type' can not be null or empty");
+ }
+ if (!CatalogUtil.isValidCatalogType(catalogType)) {
+ throw new SemanticException(String.format("type '%s' is not valid", catalogType));
Review Comment:
The CatalogUtil.isValidCatalogType method converts the input to uppercase before validation, but the test queries use both 'NATIVE' and 'native'. While this works correctly, it would be clearer to document in the checkCatalogType method that the type parameter is case-insensitive. Additionally, consider whether 'ICEBERG' catalog type requires any special handling or validation beyond just being a valid enum value.
```suggestion
/**
* Validates the mandatory catalog {@code type} property.
* <p>
* The value of {@code type} is validated in a case-insensitive manner. This method
* delegates to {@link CatalogUtil#isValidCatalogType(String)}, which normalizes the
* supplied value to upper case before checking it against the supported catalog types
* (for example, {@code NATIVE}, {@code HIVE}, {@code ICEBERG}, etc.).
* <p>
* At present, catalog types such as {@code ICEBERG} do not receive any additional
* validation in this analyzer beyond being recognized as a valid catalog type. Any
* ICEBERG-specific configuration or validation must be performed elsewhere.
*/
private static void checkCatalogType(Map<String, String> props) throws SemanticException {
String catalogType = props.get("type");
if (Strings.isNullOrEmpty(catalogType)) {
throw new SemanticException("'type' can not be null or empty");
}
if (!CatalogUtil.isValidCatalogType(catalogType)) {
// Note: validation is case-insensitive; values are normalized inside CatalogUtil.
throw new SemanticException(String.format(
"type '%s' is not valid (catalog type comparison is case-insensitive)", catalogType));
```
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat"),
Review Comment:
The documentation for METASTORE_CATALOG_WAREHOUSE has the same spacing issue. The text 'If the name of the catalog to which the db belongs is testcat,then the default path' should be 'If the name of the catalog to which the db belongs is testcat, then the default path' (note the space after the comma).
```suggestion
" then the default path prefix for the db is /user/hive/catalog/warehouse/testcat"),
```
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:
##########
@@ -388,14 +389,34 @@ public static void createDbExportDump(FileSystem fs, Path metadataPath, Database
private static void updateIfCustomDbLocations(Database database, Configuration conf) throws SemanticException {
try {
- String whLocatoion = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL,
- MetastoreConf.getVar(conf, MetastoreConf.ConfVars.WAREHOUSE));
- Path dbDerivedLoc = new Path(whLocatoion, database.getName().toLowerCase() + DATABASE_PATH_SUFFIX);
+ String catName = database.getCatalogName();
+ String dbName = database.getName().toLowerCase();
+ boolean isDefaultCatalog = Warehouse.DEFAULT_CATALOG_NAME.equals(catName);
+
+ // external warehouse root
+ String whLocation = MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE_EXTERNAL : MetastoreConf.ConfVars.WAREHOUSE_CATALOG_EXTERNAL,
+ MetastoreConf.getVar(conf,
+ isDefaultCatalog ? MetastoreConf.ConfVars.WAREHOUSE : MetastoreConf.ConfVars.WAREHOUSE_CATALOG));
+
+ if (!isDefaultCatalog) {
+ whLocation += "/" + catName;
Review Comment:
String concatenation is used to build paths instead of using the Path API. Replace 'whLocation += "/" + catName' with proper Path usage: 'whLocation = new Path(whLocation, catName).toString()' for consistency and correct path separator handling.
##########
parser/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g:
##########
@@ -1131,10 +1131,10 @@ createCatalogStatement
: KW_CREATE KW_CATALOG
ifNotExists?
name=identifier
- catLocation
+ catLocation?
catalogComment?
- (KW_PROPERTIES catprops=properties)?
- -> ^(TOK_CREATECATALOG $name catLocation ifNotExists? catalogComment? $catprops?)
+ KW_PROPERTIES catprops=properties
+ -> ^(TOK_CREATECATALOG $name catLocation? ifNotExists? catalogComment? $catprops)
Review Comment:
In the grammar change, the PROPERTIES clause is now mandatory (no '?' after 'KW_PROPERTIES catprops=properties'). However, this is a breaking change from the previous grammar where properties were optional. This means existing CREATE CATALOG statements without PROPERTIES will now fail to parse. Consider whether this breaking change is intentional or if PROPERTIES should remain optional with validation added in the analyzer to ensure the 'type' property is present.
```suggestion
(KW_PROPERTIES catprops=properties)?
-> ^(TOK_CREATECATALOG $name catLocation? ifNotExists? catalogComment? $catprops?)
```
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -1783,6 +1783,15 @@ public enum ConfVars {
"hive.metastore.warehouse.external.dir", "",
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ WAREHOUSE_CATALOG("metastore.warehouse.catalog.dir",
+ "hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
+ "location of default database for the warehouse. If the name of the catalog to which the db belongs is testcat," +
+ "then the default path prefix for the db is /user/hive/catalog/warehouse/testcat."),
Review Comment:
The description reads 'location of default database for the warehouse' which is the same as the WAREHOUSE configuration. This should be clarified to specify that this is the base location for databases in non-default catalogs. Consider rewording to: 'Base location for databases in non-default catalogs. The actual database path will be {this_value}/{catalog_name}/{database_name}.db'
```suggestion
"Base location for databases in non-default catalogs. " +
"The actual database path will be {this_value}/{catalog_name}/{database_name}.db"),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809561947
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, `NATIVE` means Hive only?
if we plan support federation won't `REMOTE` be a better choice instead of `ICEBERG`? Iceberg is a table type
@dengzhhu653, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809556549
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
Review Comment:
should this be in MetastoreConf? similar to
````
WAREHOUSE("metastore.warehouse.dir", "hive.metastore.warehouse.dir", "/user/hive/warehouse",
"location of default database for the warehouse"),
WAREHOUSE_EXTERNAL("metastore.warehouse.external.dir",
"hive.metastore.warehouse.external.dir", "",
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3749098667
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [48 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809556549
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -922,6 +922,15 @@ public static enum ConfVars {
"Default location for external tables created in the warehouse. " +
"If not set or null, then the normal warehouse location will be used as the default location."),
+ METASTORE_CATALOG_WAREHOUSE("hive.metastore.warehouse.catalog.dir", "/user/hive/catalog/warehouse",
Review Comment:
should this be in MetastoreConf? why duplicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2923472640
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
> NATIVE means Hive only?
Not quite accurate. Strictly speaking, NATIVE refers to multiple Hive-type catalogs within the same HMS. NATIVE-type catalogs belong to the same HMS.
hive type catalogs, on the other hand, are catalogs created within one HMS that point to another HMS. This means we can use a single HS2 to perform federated queries across data from multiple different HMS instances.
> if we plan to support federation won't REMOTE be a better choice instead of ICEBERG? Iceberg is a table type
For Iceberg type catalogs, I primarily refer to StarRocks' definition, where the type is set to "iceberg", and this Iceberg catalog is controlled by another parameter "iceberg.catalog.type", which can be configured as either "hive" or other types (such as rest, etc.). Here is the reference:
```
CREATE EXTERNAL CATALOG iceberg_catalog_hms
PROPERTIES
(
"type" = "iceberg",
"iceberg.catalog.type" = "hive",
"hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
"aws.s3.use_instance_profile" = "true",
"aws.s3.region" = "us-west-2"
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | ---------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4125287966
> Yes. Users can specify different locations for native catalogs; if a user does not specify a location for a native catalog, the default value will be used.
This could make some confusing. For example, we declare a S3 catalog but the location is absent, then this catalog is created with a hdfs location, some databases without explicit location will be put under the hdfs, while some are on the cloud, it might be a problem if we use the FileSystem created by the catalog's properties to operate against these tables or databases.
In HMS, we can enforce the location check when creating the catalog, if it's null and the catalog is native, then we should throw the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | -----------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | ---------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-----------------+
| Native | | Remote |
| Catalog | | Catalog |
+-------------+ +-----------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type. I assume for GLUE, there would be "glue" type, right?
- native (local),
- jdbc,
- iceberg (jdbc/hive/rest),
- hive,
- glue,
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
'type' = {'hive', 'iceberg', 'jdbc', etc}
'hive.metastore.type' = {"hive', 'glue'}
'iceberg.catalog.type' = {'hive', 'glue', 'rest'}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068005659
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
> might be. i was just thinking that when you describe catalog - we need to show the flag "EXTERNAL" = {"true", "false"} or similar
When implementing the `CREATE EXTERNAL CATALOG` syntax later, I think we can map the `EXTERNAL `keyword during catalog creation to the parameter `EXTERNAL = {"true", "false"}`, and write this parameter into the HMS metadata. **I believe this is the same principle as with external tables**. Therefore, we don't need to add an additional external parameter when creating a Hive-type catalog. I want to simplify the user's catalog creation statement as much as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068005659
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
> might be. i was just thinking that when you describe catalog - we need to show the flag "EXTERNAL" = {"true", "false"} or similar
When implementing the `CREATE EXTERNAL CATALOG` syntax later, I think we can map the `EXTERNAL `keyword during catalog creation to the parameter `EXTERNAL = {"true", "false"}`, and write this parameter into the HMS metadata. I believe this is the same principle as with external tables. Therefore, we don't need to add an additional external parameter when creating a Hive-type catalog. I want to simplify the user's catalog creation statement as much as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068019699
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
shouldn't the catalog be under the `warehousePath`?
````
Warehouse Root
└── Catalog (default or named)
└── Database (.db)
└── Table
/warehouse/<catalog_name>.cat/<database_name>.db/<table_name>/ # non default catalog
/warehouse/hive.cat/<database_name>.db/<table_name>/ # default catalog
/warehouse/hive.cat/default.db/<table_name>/ # default catalog & database
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068029854
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/create/CreateCatalogAnalyzer.java:
##########
@@ -67,10 +73,31 @@ public void analyzeInternal(ASTNode root) throws SemanticException {
}
}
+ // Initialize props if null, and set default type to native if not specified
+ if (props == null) {
+ props = new HashMap<>();
+ }
+ checkCatalogType(props);
+
CreateCatalogDesc desc = new CreateCatalogDesc(catalogName, comment, locationUrl, ifNotExists, props);
Catalog catalog = new Catalog(catalogName, locationUrl);
rootTasks.add(TaskFactory.get(new DDLWork(getInputs(), getOutputs(), desc)));
outputs.add(new WriteEntity(catalog, WriteEntity.WriteType.DDL_NO_LOCK));
}
+
+ private static void checkCatalogType(Map<String, String> props) throws SemanticException {
+ String catalogType = props.get("type");
Review Comment:
could we please create constant for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3068074140
##########
standalone-metastore/metastore-rest-catalog/src/main/java/org/apache/iceberg/rest/HMSCatalogFactory.java:
##########
@@ -79,6 +79,7 @@ private Catalog createCatalog() {
if (configUri != null && !configUri.isEmpty()) {
properties.put("uri", configUri);
}
+ // TODO catalog. Consider adding new created native catalog warehouse(MetastoreConf.ConfVars.WAREHOUSE_CATALOG) later.
Review Comment:
could you please elaborate, i don't quite get what you are suggesting here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3122250914
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import com.google.common.base.Strings;
+
+public class CatalogUtil {
+ public static final String TYPE = "type";
+
+ // Supported Catalog types
+ public enum CatalogType {
+ HIVE,
Review Comment:
I don't quite understand what you mean. Could you explain it in more detail? Thank you.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -694,6 +694,9 @@ private void rollbackTransactionToSavePoint(String savePoint) {
@Override
public void createCatalog(Catalog cat) throws MetaException {
LOG.debug("Creating catalog {}", cat);
+
+ ensureCatalogType(cat);
Review Comment:
`HMSHandler#create_catalog` may not be the only entry point for creating a catalog. So I think it would be more appropriate to place it in ObjectStore.
What negative impacts do you think putting it here might have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2992207451
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
I like the Gravitino idea. Catalog is a set of metadata describing the source, so it's good for managing and discovery. It should answer two questions from Metastore's point of view: Where is catalog and what is Catalog.
Where:
Native, as we are positioned today, this catalog is inside the Metastore.
Remote(Foreign), this catalog is outside of this Metastore.
What:
The catalog describing could be a jdbc, hbase, and iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
"type" = {"hive", "iceberg", "jdbc", etc}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3763957612
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [63 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-3764989109
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267) **Quality Gate passed**
Issues
 [63 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6267&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6267&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6267&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6267)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2809561947
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, `NATIVE` means Hive only?
if we plan support federation won't `REMOTE` be a better choice instead of `ICEBERG`? Iceberg is a table type
FYI: https://polaris.apache.org/in-dev/unreleased/federation/hive-metastore-federation/
@dengzhhu653, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2975646270
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
IMO it sounds more like connector type, not catalog type.
- native (local) - that's what we have now, with external, acid and iceberg tables support;
- jdbc,
- iceberg (jdbc/hive/rest),
- hive,
- glue
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4110220860
> One thing to note: Currently, we have two entry points for creating catalogs—one is the SQL-based catalog creation entry involved in this PR, and the other is the HMS API-based catalog creation entry. In this PR, I intend to enforce the inclusion of the `type` attribute when users create catalogs via SQL, so that users can clearly distinguish between different types of catalogs. However, the HMS API side does not enforce the `type` attribute when creating catalogs, which would lead to inconsistency between the two entry points.
>
> Therefore, I have two approaches to resolve this issue and ensure consistency between the two entry points:
>
> 1. On the HMS server side, specifically in the `HMSHandler`, enforce a check during catalog creation to verify whether the user has provided the `type` attribute. If the `type` is missing, return an error. This approach would introduce backward compatibility issues for catalog creation.
> 2. Modify the current PR to make the `type` attribute optional for SQL-based catalog creation. For both SQL-based catalog creation and HMS API-based catalog creation, if the user does not provide the `type` attribute, assign a default value (e.g., `native`) on the HMSHandler server side. This approach would not cause compatibility issues with the catalog creation syntax. However, other open-source components enforce the `type` attribute requirement, and the `type` helps users better understand the parameter requirements for the catalog, among other benefits.
>
> What are your suggestions regarding these two approaches? Thanks in advance. cc @deniskuzZ @difin
i would vote for option 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +-------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +--------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +--------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +---------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2974740820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
@zhangbutao, doest the below sketch reflect what you have in mind? What if we adopt hoodi later?
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-------------------------------------------------
| | |
+-------------+ +----------------+ +-----------------+
| Native | | Hive Remote | | Iceberg |
| Catalog | | Catalog | | Catalog |
+-------------+ +----------------+ +-----------------+
| Same HMS | | Remote HMS | | Backend varies: |
| Hive tables | | Hive tables | | Hive / REST / |
| Only local | | Federation | | HDFS |
+-------------+ +----------------+ | Tables in |
| Iceberg format |
+-----------------+
````
isn't it enough to have just 2 categories
````
+---------------------+
| StarRocks / HS2 |
+---------------------+
|
-----------------------------------
| |
+-------------+ +------------------+
| Native | | Remote / External |
| Catalog | | Catalog |
+-------------+ +-------------------+
| Local HMS | |
| Hive tables | --------------------------------------------
| Iceberg | | | |
| tables | +-----------------+ +-----------------+ +-----------------+
+-------------+ | External HMS | | REST Catalog | | GUE Catalog |
| Hive tables | | Iceberg tables | | Iceberg tables |
| Iceberg tables | +-----------------+ +-----------------+
+-----------------+
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #6267:
URL: https://github.com/apache/hive/pull/6267#issuecomment-4125053032
> > tables and databases under the newly created native catalog do not respect its own catalog location.
>
> Let's address it separately, check the catalog's location when creating the database.
>
> I don't like assuming we must have a location in Catalog, for a non-native catalog(Let's rename it to Foreign catalog), such as JDBC, Kudu, Kafka(the catalog type), this location is unnecessary IMO. It doesn't mean every native catalog should reside under the same parent directory or file system. For example, in a hybrid env the catalog directory could be a S3 bucket or a hdfs directory.
> I don't like assuming we must have a location in Catalog, for a non-native catalog(Let's rename it to Foreign catalog), such as JDBC, Kudu, Kafka(the catalog type), this location is unnecessary IMO.
Yes. I've stated in the documentation that only native catalogs require a location. Other catalogs do not.
> It doesn't mean every native catalog should reside under the same parent directory or file system.
Yes. Users can specify different locations for native catalogs; if a user does not specify a location for a native catalog, the default value will be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
"type" = {"hive", "iceberg", "jdbc", "rest", etc}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "EXTERNAL" = {"true" , "false"}
'type' = {'hive', 'iceberg', 'jdbc', etc}
'hive.metastore.type' = {'hive', 'glue'}, maybe 'rest' in future
'iceberg.catalog.type' = {'hive', 'glue', 'rest'}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "EXTERNAL" = {"true" , "false"}
'type' = {'hive', 'iceberg', 'jdbc', etc}
'hive.metastore.type' = {'hive', 'glue'}
'iceberg.catalog.type' = {'hive', 'glue', 'rest'}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r2994451977
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CatalogUtil.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.EnumUtils;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public class CatalogUtil {
+ public enum CatalogType {
+ NATIVE,
Review Comment:
+1 on Zhihua's suggestion
`CREATE EXTERNAL CATALOG` - Foreign catalog
`CREATE CATALOG` - Native catalog
under the hood we can add propery "native" = {"true" , "false"}
"type" = {"hive", "iceberg", "jdbc", etc}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
Sorry, I might be missing something. Where is the confusion [here]? Could you please elaborate (https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
I'm aligned on this as well
+1
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
if user doesn't explicitly set the catalog location, why not use default set by `metastore.warehouse.catalog.dir` ?
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29241: Distinguish the default location of the database by catalog [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6267:
URL: https://github.com/apache/hive/pull/6267#discussion_r3137343191
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMiniClusters.java:
##########
@@ -554,23 +556,27 @@ private void setFsRelatedProperties(HiveConf conf, boolean isLocalFs, FileSystem
// Create a fake fs root for local fs
Path localFsRoot = new Path(path, "localfs");
warehousePath = new Path(localFsRoot, "warehouse");
+ warehouseCatPath = new Path(localFsRoot, "catalog");
Review Comment:
@zhangbutao
> The warehousePath is the path of the default catalog, under which the paths of Hive databases reside. If the catalog path is placed under the warehousePath, it will cause confusion with the path information of Hive databases under the default catalog.
i don't get that at all. where is confusion [here](https://github.com/apache/hive/pull/6267#discussion_r3068019699)?
> The path for non-default catalogs is controlled by the new added property metastore.warehouse.catalog.dir
i don't think we need additional config for that
@dengzhhu653
> I don't like to add a new metastore.warehouse.catalog.dir as a default location for new catalogs(non-default). this could bring some concerns IMO
I'm aligned on this as well
+1
> On creating the catalog, we create the default database as well, the default database will stay at the old location even if we change the location of the catalog
default DB location should be under it's catalog location IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org