[GitHub] [hive] kasakrisz commented on a change in pull request #2991: HIVE-25918: Invalid stats after multi inserting into the same partition

[GitBox <gi...@apache.org>]

kasakrisz commented on a change in pull request #2991:
URL: https://github.com/apache/hive/pull/2991#discussion_r800706374



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java
##########
@@ -512,6 +513,8 @@ private String toString(Map<String, String> parameters) {
         if (dpPartSpecs != null) {
           // load the list of DP partitions and return the list of partition specs
           list.addAll(dpPartSpecs);
+          // Reload partition metadata because another BasicStatsTask instance may have updated the stats.
+          list = db.getPartitionsByNames(table, list.stream().map(Partition::getName).collect(Collectors.toList()));

Review comment:
       I think these are thrift API calls from HS2 to HMS and the number of calls depend on the number of partitions and `hive.metastore.batch.retrieve.max`.
   Also there is some communication from HMS to backend db which is also done in batches.
   
   I explored another way:
   * In the StatsTask instead of applying the stats on the Partition objects collect it to a delta objects.
   * Implement a HMS call and backend DB call to update stats based on the collected delta objects.
   However this would be a bigger change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org