[GitHub] [hive] pvary commented on a diff in pull request #3053: HIVE-25980: Reduce fs calls in HiveMetaStoreChecker.checkTable

[GitBox <gi...@apache.org>]

pvary commented on code in PR #3053:
URL: https://github.com/apache/hive/pull/3053#discussion_r888121662


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java:
##########
@@ -308,8 +308,38 @@ void checkTable(Table table, PartitionIterable parts, byte[] filterExp, CheckRes
       result.getTablesNotOnFs().add(table.getTableName());
       return;
     }
+    
+    // now check the table folder and see if we find anything
+    // that isn't in the metastore
+    Set<Path> allPartDirs = new HashSet<>();
+    List<FieldSchema> partColumns = table.getPartitionKeys();
+    checkPartitionDirs(tablePath, allPartDirs, Collections.unmodifiableList(getPartColNames(table)));
+
+    if (filterExp != null) {
+      PartitionExpressionProxy expressionProxy = createExpressionProxy(conf);
+      List<String> partitions = new ArrayList<>();
+      Set<Path> partDirs = new HashSet<Path>();
+      String tablePathStr = tablePath.toString();
+      for (Path path : allPartDirs) {

Review Comment:
   Could we do this with Stream, and filter and friends?
   Also we should calculate `tablePathStr.endsWith("/")` only once



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org