[GitHub] [hive] rbalamohan commented on a change in pull request #2520: HIVE-25373 : Modify buildColumnStatsDesc to send configured number of stats for updation

[GitBox <gi...@apache.org>]

rbalamohan commented on a change in pull request #2520:
URL: https://github.com/apache/hive/pull/2520#discussion_r677960313



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java
##########
@@ -177,25 +182,36 @@ private ColumnStatisticsDesc buildColumnStatsDesc(Table table, String partName,
   public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaException, IOException {
     // Construct a column statistics object from the result
 
-    List<ColumnStatistics> colStats = constructColumnStatsFromPackedRows(tbl);
-    // Persist the column statistics object to the metastore
-    // Note, this function is shared for both table and partition column stats.
-    if (colStats.isEmpty()) {
-      return 0;
-    }
-    SetPartitionsStatsRequest request = new SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
-    request.setNeedMerge(colStatDesc.isNeedMerge());
-    HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl)
-        ? SessionState.get().getTxnMgr() : null;
+    long writeId = -1;
+    ValidWriteIdList validWriteIdList = null;
+    HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl) ? SessionState.get().getTxnMgr() : null;
     if (txnMgr != null) {
-      request.setWriteId(txnMgr.getAllocatedTableWriteId(tbl.getDbName(), tbl.getTableName()));
-      ValidWriteIdList validWriteIdList =
-          AcidUtils.getTableValidWriteIdList(conf, AcidUtils.getFullTableName(tbl.getDbName(), tbl.getTableName()));
-      if (validWriteIdList != null) {
-        request.setValidWriteIdList(validWriteIdList.toString());
+      writeId = txnMgr.getAllocatedTableWriteId(tbl.getDbName(), tbl.getTableName());
+      validWriteIdList =
+              AcidUtils.getTableValidWriteIdList(conf, AcidUtils.getFullTableName(tbl.getDbName(), tbl.getTableName()));
+    }
+
+    boolean done = false;
+    long maxNumStats = conf.getLongVar(HiveConf.ConfVars.HIVE_STATS_MAX_NUM_STATS);
+    while (!done) {
+      List<ColumnStatistics> colStats = new ArrayList<>();
+      done = constructColumnStatsFromPackedRows(tbl, colStats, maxNumStats);
+
+      // Persist the column statistics object to the metastore
+      // Note, this function is shared for both table and partition column stats.
+      if (colStats.isEmpty()) {
+        continue;
+      }
+      SetPartitionsStatsRequest request = new SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
+      request.setNeedMerge(colStatDesc.isNeedMerge());
+      if (txnMgr != null) {
+        request.setWriteId(writeId);
+        if (validWriteIdList != null) {
+          request.setValidWriteIdList(validWriteIdList.toString());
+        }
       }
+      db.setPartitionColumnStatistics(request);

Review comment:
       Info level logging would be good to have here. Since it will be printed for every 10K updates, it should be fine and will be helpful for debugging as well. And if possible provide time taken for overall buildColumnStatsDesc.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org