Re: [PR] HIVE-28813: MSCK/Analyze commands can show a warning in console for Small files. [hive]

["jameskramden (via GitHub)" <gi...@apache.org>]

jameskramden commented on code in PR #5950:
URL: https://github.com/apache/hive/pull/5950#discussion_r2314159271


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -358,6 +361,28 @@ void checkTable(Table table, PartitionIterable parts, byte[] filterExp, CheckRes
       if (partPath == null) {
         continue;
       }
+
+      // check the table/partition if (totalSize / numFiles) is less than avgFileSize
+      Map<String, String> partitionParameters = partition.getParameters();
+      if (partitionParameters != null) {
+        long totalSize = Long.parseLong(partitionParameters.getOrDefault(StatsSetupConst.TOTAL_SIZE, "0"));
+        long numFiles = Long.parseLong(partitionParameters.getOrDefault(StatsSetupConst.NUM_FILES, "0"));
+        if (numFiles != 0) {
+          long avgFileSize = totalSize / numFiles;
+          long HIVE_FILES_AVG_SIZE = conf.getLong("hive.merge.smallfiles.avgsize", 0);
+          if (avgFileSize <= HIVE_FILES_AVG_SIZE) {
+            StringBuilder tmpStatsSb = new StringBuilder();
+            tmpStatsSb.append("totalSize = ").append(totalSize).append(", numFiles = ")
+                    .append(numFiles).append(". ");
+            smallFilesStats.putIfAbsent(Warehouse.makePartName(table.getPartitionKeys(), partition.getValues()), tmpStatsSb.toString());
+            result.setSmallFilesStats(smallFilesStats);
+          }
+        } else {
+          LOG.warn("Partition total number of files is 0.");

Review Comment:
   This line logs a warning message that a partition has 0 files. This is bad because it doesn't add information which partition has 0 files. Log may contain several warnings that a partition has 0 files and the person who reviews the log won't know which partitions are empty.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org