[GitHub] [hive] rbalamohan commented on a change in pull request #2991: HIVE-25918: Invalid stats after multi inserting into the same partition

[GitBox <gi...@apache.org>]

rbalamohan commented on a change in pull request #2991:
URL: https://github.com/apache/hive/pull/2991#discussion_r801136555



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java
##########
@@ -512,6 +513,8 @@ private String toString(Map<String, String> parameters) {
         if (dpPartSpecs != null) {
           // load the list of DP partitions and return the list of partition specs
           list.addAll(dpPartSpecs);
+          // Reload partition metadata because another BasicStatsTask instance may have updated the stats.
+          list = db.getPartitionsByNames(table, list.stream().map(Partition::getName).collect(Collectors.toList()));

Review comment:
       getPartitionByNames is a single call unlike 2000+ getPartition() calls (Which internally loads table object everytime etc). It will still add up to runtime depending on the number of partitions for regular insert queries.
   
   One option could be to figure out if it is multi-insert at the time of sem analyzer and pass it on to DynamicPartitionCtx; getPartitionByNames() could be invoked only in those conditions.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org