You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/10/18 19:53:18 UTC
[GitHub] [hive] nssalian opened a new pull request #2730: Hive-24741 backport to 2.3
nssalian opened a new pull request #2730:
URL: https://github.com/apache/hive/pull/2730
### **What changes were proposed in this pull request?**
This PR is a backport of https://github.com/apache/hive/pull/1948 wherein the the performance of get_partitions_ps_with_auth is improved into the Hive-2.3 branch.
### **Why are the changes needed?**
Performance improvement for a HMS API on the Hive-2.3 branch since Spark continues to use Hive 2.3.x.
### **Does this PR introduce any user-facing change?**
No
### **How was this patch tested?**
Same tests as original patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nssalian commented on pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #2730:
URL: https://github.com/apache/hive/pull/2730#issuecomment-946957642
Thanks @vihangk1 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 commented on a change in pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 commented on a change in pull request #2730:
URL: https://github.com/apache/hive/pull/2730#discussion_r731302621
##########
File path: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -2565,34 +2566,68 @@ private Collection getPartitionPsQueryResults(String dbName, String tableName,
return (Collection) query.execute(dbName, tableName, partNameMatcher);
}
+ /**
+ * If partVals all the values are empty strings, it means we are returning
+ * all the partitions and hence we can attempt to use a directSQL equivalent API which
+ * is considerably faster.
+ * @param partVals The partitions values used to filter out the partitions.
+ * @return true if partVals is empty or if all the values in partVals is empty strings.
Review comment:
I know this is from the original patch but looks like this javadoc is incorrect. The method actually returns false if partVals is empty. I don't really remember now why that was the case. Perhaps you can look at this if you have some cycles? If not, please fix the java doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 commented on pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 commented on pull request #2730:
URL: https://github.com/apache/hive/pull/2730#issuecomment-947053721
I am overriding the tests failed label in this case. Hive 2.x is a pretty old branch. We don't have a baseline of how stable are the tests on that branch in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 commented on a change in pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 commented on a change in pull request #2730:
URL: https://github.com/apache/hive/pull/2730#discussion_r731326469
##########
File path: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -2571,8 +2571,8 @@ private Collection getPartitionPsQueryResults(String dbName, String tableName,
* all the partitions and hence we can attempt to use a directSQL equivalent API which
* is considerably faster.
* @param partVals The partitions values used to filter out the partitions.
- * @return true if partVals is empty or if all the values in partVals is empty strings.
- * other wise false. If user or groups is valid then returns false since the directSQL
+ * @return false if partVals is empty, true if all the values in partVals are empty strings and
Review comment:
the comment should be "...and otherwise false for all the other cases" Or more simply, the method returns true only when partVals is non-empty and contains only empty strings, otherwise false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 commented on pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 commented on pull request #2730:
URL: https://github.com/apache/hive/pull/2730#issuecomment-947055116
Thanks @nssalian for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 commented on pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 commented on pull request #2730:
URL: https://github.com/apache/hive/pull/2730#issuecomment-946956766
Thanks for addressing the changes. I don't think the precommits are stable for Hive 2.x branch so I am willing to override testing results and do a force merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nssalian commented on a change in pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #2730:
URL: https://github.com/apache/hive/pull/2730#discussion_r731331838
##########
File path: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -2571,8 +2571,8 @@ private Collection getPartitionPsQueryResults(String dbName, String tableName,
* all the partitions and hence we can attempt to use a directSQL equivalent API which
* is considerably faster.
* @param partVals The partitions values used to filter out the partitions.
- * @return true if partVals is empty or if all the values in partVals is empty strings.
- * other wise false. If user or groups is valid then returns false since the directSQL
+ * @return false if partVals is empty, true if all the values in partVals are empty strings and
Review comment:
makes sense. Fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] vihangk1 merged pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
vihangk1 merged pull request #2730:
URL: https://github.com/apache/hive/pull/2730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nssalian commented on pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #2730:
URL: https://github.com/apache/hive/pull/2730#issuecomment-946130817
CC: @nrg4878 , @vihangk1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nssalian commented on a change in pull request #2730: HIVE-25616: Hive-24741 backport to 2.3
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #2730:
URL: https://github.com/apache/hive/pull/2730#discussion_r731308987
##########
File path: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##########
@@ -2565,34 +2566,68 @@ private Collection getPartitionPsQueryResults(String dbName, String tableName,
return (Collection) query.execute(dbName, tableName, partNameMatcher);
}
+ /**
+ * If partVals all the values are empty strings, it means we are returning
+ * all the partitions and hence we can attempt to use a directSQL equivalent API which
+ * is considerably faster.
+ * @param partVals The partitions values used to filter out the partitions.
+ * @return true if partVals is empty or if all the values in partVals is empty strings.
Review comment:
Will fix the docstring
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org