You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/07/15 23:00:44 UTC
[GitHub] [hive] hankfanchiu opened a new pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
hankfanchiu opened a new pull request #2484:
URL: https://github.com/apache/hive/pull/2484
### What changes were proposed in this pull request?
On the Hive MetaStore server's side of the `HiveMetaStoreClient#listTableNamesByFilter()` API, conditionally manipulate the string filter by replacing any `=` comparison on a table parameter with a `LIKE` operator.
This replacement is only performed if:
1. The database product backing the Hive MetaStore is Derby,
2. The `=` comparison is on a table parameter, i.e. the `"TABLE_PARAMS"."PARAM_VALUE"` column.
### Why are the changes needed?
HIVE-12274 changed the type of the `"TABLE_PARAMS"."PARAM_VALUE"` column to `CLOB` for Derby and Oracle, e.g.:
https://github.com/apache/hive/blob/7d4134e7fe9bfb8b8aa3344a9ae72f4f36c98b2c/metastore/scripts/upgrade/derby/039-HIVE-12274.derby.sql#L8-L12
With the type change, invoking `ObjectStore#listTableNamesByFilter()` given a `=` comparison -- on a table parameter stored in Derby -- fails with the following exception:
```
ERROR 42818: Comparisons between 'CLOB (UCS_BASIC)' and 'CLOB (UCS_BASIC)' are not supported. Types must be comparable. String types must also have matching collation. If collation does not match, a possible solution is to cast operands to force them to the default collation (e.g. SELECT tablename FROM sys.systables WHERE CAST(tablename AS VARCHAR(128)) = 'T1')
```
Without reverting the column type change, we add support for `=` comparison on table parameters in Derby.
### Does this PR introduce _any_ user-facing change?
Yes, invoking `HiveMetaStoreClient#listTableNamesByFilter()` with a `$param_key = '$param_value'` filter now succeeds for Derby databases.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2484:
URL: https://github.com/apache/hive/pull/2484#discussion_r671113322
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -4693,6 +4694,17 @@ private Table ensureGetTable(String catName, String dbName, String tblName)
return convertToTable(ensureGetMTable(catName, dbName, tblName));
}
+ /**
+ * Conditionally alters an {@link ExpressionTree} to be compatible with certain database products.
+ */
+ private void reconcileTreeWithDatabaseProduct(ExpressionTree tree) throws MetaException {
+ if (sqlGenerator.getDbProduct().isDERBY()) {
Review comment:
Is this only problem for Derby?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2484:
URL: https://github.com/apache/hive/pull/2484#issuecomment-974730544
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hankfanchiu commented on pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
hankfanchiu commented on pull request #2484:
URL: https://github.com/apache/hive/pull/2484#issuecomment-881063844
@pvary, mind taking a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hankfanchiu commented on a change in pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
hankfanchiu commented on a change in pull request #2484:
URL: https://github.com/apache/hive/pull/2484#discussion_r671413592
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -4693,6 +4694,17 @@ private Table ensureGetTable(String catName, String dbName, String tblName)
return convertToTable(ensureGetMTable(catName, dbName, tblName));
}
+ /**
+ * Conditionally alters an {@link ExpressionTree} to be compatible with certain database products.
+ */
+ private void reconcileTreeWithDatabaseProduct(ExpressionTree tree) throws MetaException {
+ if (sqlGenerator.getDbProduct().isDERBY()) {
Review comment:
@findepi mentioned in https://issues.apache.org/jira/browse/HIVE-21614?focusedCommentId=16849571&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16849571 that Oracle may be implicated as well, but I have not verified.
Having this `ObjectStore#reconcileTreeWithDatabaseProduct` defined allows us to conveniently add other databases, though, so perhaps we could revisit as needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2484:
URL: https://github.com/apache/hive/pull/2484#discussion_r671114558
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/OperatorEqualsToLikeReplacer.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.metastore.parser.ExpressionTree.LeafNode;
+import org.apache.hadoop.hive.metastore.parser.ExpressionTree.Operator;
+import org.apache.hadoop.hive.metastore.parser.ExpressionTree.TreeVisitor;
+
+/**
+ * Replaces any {@link Operator.EQUALS} with {@link Operator.LIKE} for {@link LeafNode}
+ * representing an equality comparison on a table parameter.
+ *
+ * This utility is effective for database products that do not support equality comparisons
+ * between values of the CLOB type.
+ */
+final class OperatorEqualsToLikeReplacer extends TreeVisitor {
+ @Override
+ protected void visit(LeafNode node) throws MetaException {
+ if (
Review comment:
nit: Could we format this way:
```
if (node.keyName.startsWith(hive_metastoreConstants.HIVE_FILTER_FIELD_PARAMS)
&& node.operator == Operator.EQUALS) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2484:
URL: https://github.com/apache/hive/pull/2484#issuecomment-974730544
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] hankfanchiu commented on pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
hankfanchiu commented on pull request #2484:
URL: https://github.com/apache/hive/pull/2484#issuecomment-897122168
@pvary, is this good to go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #2484:
URL: https://github.com/apache/hive/pull/2484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #2484: HIVE-21614: "Support" comparing Derby's table parameter CLOB by replacing `=` with `LIKE`
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #2484:
URL: https://github.com/apache/hive/pull/2484#discussion_r691825714
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -4693,6 +4694,17 @@ private Table ensureGetTable(String catName, String dbName, String tblName)
return convertToTable(ensureGetMTable(catName, dbName, tblName));
}
+ /**
+ * Conditionally alters an {@link ExpressionTree} to be compatible with certain database products.
+ */
+ private void reconcileTreeWithDatabaseProduct(ExpressionTree tree) throws MetaException {
+ if (sqlGenerator.getDbProduct().isDERBY()) {
Review comment:
There are Docker images for the supported databases. Could you please run tests with them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org