You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/17 22:31:39 UTC

[GitHub] [hudi] prashantwason opened a new pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

prashantwason opened a new pull request #2343:
URL: https://github.com/apache/hudi/pull/2343


   This finds partitions and files in a single scan rather than listing partitions first and then listing each partition.
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   mvn-test -pl hudi-client -Dtest=TestHoodieFsMetadata
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io edited a comment on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (374bbb6) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/2e09aeca47a7f14cd83f644a5132d4dfd0f27813?el=desc) (2e09aec) will **decrease** coverage by `15.55%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             rfc-15    #2343       +/-   ##
   =============================================
   - Coverage     59.44%   43.88%   -15.56%     
   + Complexity     4592     3379     -1213     
   =============================================
     Files           573      573               
     Lines         24353    24362        +9     
     Branches       2440     2440               
   =============================================
   - Hits          14476    10692     -3784     
   - Misses         8825    12654     +3829     
   + Partials       1052     1016       -36     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `27.15% <0.00%> (-39.54%)` | `0.00 <0.00> (ø)` | |
   | hudiclient | `24.34% <0.00%> (-51.02%)` | `0.00 <0.00> (ø)` | |
   | hudicommon | `52.78% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudihadoopmr | `33.05% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudispark | `67.19% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | huditimelineservice | `64.43% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudiutilities | `69.38% <ø> (ø)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `2.16% <0.00%> (-68.76%)` | `2.00 <0.00> (-48.00)` | |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Ib29kaWVHYXVnZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-24.00%)` | |
   | [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...g/apache/hudi/exception/HoodieUpsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZVVwc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllS2V5TG9jYXRpb25GZXRjaEhhbmRsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [...apache/hudi/exception/HoodieRollbackException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZVJvbGxiYWNrRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | ... and [137 more](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] umehrot2 commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r545546646



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,

Review comment:
       - Perhaps name it `getPartitionsToFilesMapping()` ?
   - Ideally, it would have been good to have it as a utility in `FsUtils` but because of our current structure we can't move it, since `FsUtils` is in `hudi-common` which we don't want to depend on `spark`.

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))
+            .map(path -> {
+              FileSystem fs = datasetMetaClient.getFs();

Review comment:
       This adds unnecessary overhead of serialization/deserialization of `HoodieTableMetaClient`. Instead, we can  do:
   ```
   path.getFileSystem(new Configuration());
   ```
   to get the file system.

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))

Review comment:
       I think we should introduce like a `LISTING_PARALLELISM` property with a default of `1500` and use it here. (https://github.com/apache/hudi/blob/rfc-15/hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java#L70)

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))
+            .map(path -> {
+              FileSystem fs = datasetMetaClient.getFs();
+              return Pair.of(path, fs.listStatus(path));
+            }).collect();
+      pathsToList.clear();
+
+      // If the listing reveals a directory, add it to queue. If the listing reveals a hoodie partition, add it to
+      // the results.
+      dirToFileListing.forEach(p -> {
+        List<FileStatus> filesInDir = Arrays.stream(p.getRight())
+            .filter(fs -> !fs.getPath().getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))
+            .collect(Collectors.toList());

Review comment:
       May be `Arrays.stream().parallel()` ? This can be a huge list to go through for large datasets.

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -51,6 +51,7 @@
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;

Review comment:
       Can you re-use this same thing to `getAllPartitionPaths()` here: https://github.com/apache/hudi/blob/rfc-15/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java#L149
   
   The default mechanism is really slow. I investigated `FsUtils.getAllPartitionPaths()` is also used at multiple places across Hudi code-base which can benefit from RFC-15. So, what I am thinking is by default we can let `FsUtils.getAllPartitionPaths()` always default to the metadata table, and internally if the table is not present, it will use this default parallelized/optimized listing of partition paths.
   
   So, it benefits both customers who use metadata table and who don't.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r546119001



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -51,6 +51,7 @@
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;

Review comment:
       I wanted to do that but noticed that HoodieBackedTableMetadata is in hudi-common which does not have spark.
   
   Some new interface / refactoring will be required. Can handle this in a different ticket.

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))
+            .map(path -> {
+              FileSystem fs = datasetMetaClient.getFs();
+              return Pair.of(path, fs.listStatus(path));
+            }).collect();
+      pathsToList.clear();
+
+      // If the listing reveals a directory, add it to queue. If the listing reveals a hoodie partition, add it to
+      // the results.
+      dirToFileListing.forEach(p -> {
+        List<FileStatus> filesInDir = Arrays.stream(p.getRight())
+            .filter(fs -> !fs.getPath().getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))
+            .collect(Collectors.toList());

Review comment:
       Done. 
   

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))

Review comment:
       Done

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r546416432



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -51,6 +51,7 @@
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;

Review comment:
       https://issues.apache.org/jira/browse/HUDI-1479 filed to track follow up. This needs to be done on master, since it has `HoodieEngineContext` abstraction that can be moved to hudi-common. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io edited a comment on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (d71922d) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/c6c7e18701b4d5a9435451b537b973be05342511?el=desc) (c6c7e18) will **decrease** coverage by `37.60%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             rfc-15    #2343       +/-   ##
   =============================================
   - Coverage     59.25%   21.65%   -37.61%     
   + Complexity     4591      561     -4030     
   =============================================
     Files           574      252      -322     
     Lines         24385     9120    -15265     
     Branches       2442      828     -1614     
   =============================================
   - Hits          14450     1975    -12475     
   + Misses         8884     6879     -2005     
   + Partials       1051      266      -785     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `?` | `?` | |
   | hudiclient | `24.25% <0.00%> (-51.10%)` | `0.00 <0.00> (ø)` | |
   | hudicommon | `?` | `?` | |
   | hudihadoopmr | `?` | `?` | |
   | hudispark | `?` | `?` | |
   | huditimelineservice | `?` | `?` | |
   | hudiutilities | `10.53% <ø> (-58.86%)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `39.40% <0.00%> (-41.27%)` | `65.00 <0.00> (-49.00)` | |
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `2.14% <0.00%> (-68.85%)` | `2.00 <0.00> (-48.00)` | |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Ib29kaWVHYXVnZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-24.00%)` | |
   | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
   | [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | ... and [494 more](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io edited a comment on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (b7fc074) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/2e09aeca47a7f14cd83f644a5132d4dfd0f27813?el=desc) (2e09aec) will **decrease** coverage by `37.79%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             rfc-15    #2343       +/-   ##
   =============================================
   - Coverage     59.44%   21.64%   -37.80%     
   + Complexity     4592      560     -4032     
   =============================================
     Files           573      252      -321     
     Lines         24353     9119    -15234     
     Branches       2440      828     -1612     
   =============================================
   - Hits          14476     1974    -12502     
   + Misses         8825     6879     -1946     
   + Partials       1052      266      -786     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `?` | `?` | |
   | hudiclient | `24.24% <0.00%> (-51.13%)` | `0.00 <0.00> (ø)` | |
   | hudicommon | `?` | `?` | |
   | hudihadoopmr | `?` | `?` | |
   | hudispark | `?` | `?` | |
   | huditimelineservice | `?` | `?` | |
   | hudiutilities | `10.53% <ø> (-58.86%)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `39.10% <0.00%> (-41.57%)` | `64.00 <0.00> (-50.00)` | |
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `2.14% <0.00%> (-68.77%)` | `2.00 <0.00> (-48.00)` | |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Ib29kaWVHYXVnZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-24.00%)` | |
   | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
   | [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | ... and [493 more](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747747412


   Good optimization. Will review 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] umehrot2 commented on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747755598


   Its much needed. We should have done this for Hudi in general long time back.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
prashantwason commented on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747741409


   @vinothchandar  Please take a look.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io edited a comment on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (374bbb6) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/2e09aeca47a7f14cd83f644a5132d4dfd0f27813?el=desc) (2e09aec) will **decrease** coverage by `0.00%`.
   > The diff coverage is `90.32%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             rfc-15    #2343      +/-   ##
   ============================================
   - Coverage     59.44%   59.44%   -0.01%     
   - Complexity     4592     4598       +6     
   ============================================
     Files           573      573              
     Lines         24353    24362       +9     
     Branches       2440     2440              
   ============================================
   + Hits          14476    14481       +5     
   - Misses         8825     8831       +6     
   + Partials       1052     1050       -2     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `66.67% <90.32%> (-0.02%)` | `0.00 <11.00> (ø)` | |
   | hudiclient | `75.34% <90.32%> (-0.03%)` | `0.00 <11.00> (ø)` | |
   | hudicommon | `52.78% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudihadoopmr | `33.05% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudispark | `67.19% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | huditimelineservice | `64.43% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudiutilities | `69.38% <ø> (ø)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `72.70% <90.32%> (+1.78%)` | `59.00 <11.00> (+9.00)` | |
   | [...a/org/apache/hudi/metrics/DistributedRegistry.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9EaXN0cmlidXRlZFJlZ2lzdHJ5LmphdmE=) | `63.33% <0.00%> (-30.01%)` | `9.00% <0.00%> (-4.00%)` | |
   | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllU29ydGVkTWVyZ2VIYW5kbGUuamF2YQ==) | `52.50% <0.00%> (+2.50%)` | `6.00% <0.00%> (+1.00%)` | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r545976491



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -51,6 +51,7 @@
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;

Review comment:
       >So, what I am thinking is by default we can let FsUtils.getAllPartitionPaths() always default to the metadata table, and internally if the table is not present, it will use this default parallelized/optimized listing of partition paths.
   
   +1 on this. There is a JIRA to track this specifically




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io edited a comment on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (d71922d) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/c6c7e18701b4d5a9435451b537b973be05342511?el=desc) (c6c7e18) will **increase** coverage by `3.81%`.
   > The diff coverage is `92.10%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             rfc-15    #2343      +/-   ##
   ============================================
   + Coverage     59.25%   63.07%   +3.81%     
   + Complexity     4591     1779    -2812     
   ============================================
     Files           574      252     -322     
     Lines         24385     9120   -15265     
     Branches       2442      828    -1614     
   ============================================
   - Hits          14450     5752    -8698     
   + Misses         8884     3062    -5822     
   + Partials       1051      306     -745     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `?` | `?` | |
   | hudiclient | `75.35% <92.10%> (-0.01%)` | `0.00 <12.00> (ø)` | |
   | hudicommon | `?` | `?` | |
   | hudihadoopmr | `?` | `?` | |
   | hudispark | `?` | `?` | |
   | huditimelineservice | `?` | `?` | |
   | hudiutilities | `10.53% <ø> (-58.86%)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `72.92% <91.17%> (+1.92%)` | `59.00 <11.00> (+9.00)` | |
   | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `80.89% <100.00%> (+0.23%)` | `115.00 <1.00> (+1.00)` | |
   | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
   | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
   | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
   | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
   | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
   | ... and [352 more](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r546313057



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -72,6 +72,7 @@
   public static final String BULKINSERT_USER_DEFINED_PARTITIONER_CLASS = "hoodie.bulkinsert.user.defined.partitioner.class";
   public static final String UPSERT_PARALLELISM = "hoodie.upsert.shuffle.parallelism";
   public static final String DELETE_PARALLELISM = "hoodie.delete.shuffle.parallelism";
+  public static final String FILE_LISTING_PARALLELISM = "hoodie.file.listing.shuffle.parallelism";

Review comment:
       I think we can drop `.shuffle` 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -51,6 +51,7 @@
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;

Review comment:
       We can pass in the `HoodieEngineContext` which can also be plain old java if needed. I ll take a closer look in #2351 and how best we can do this. We can scope this PR to just what you just fixed now




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#discussion_r546120188



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -369,10 +343,56 @@ private void bootstrapFromFilesystem(JavaSparkContext jsc, HoodieTableMetaClient
       }
     });
 
-    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    LOG.info("Committing " + partitionToFileStatus.size() + " partitions and " + stats[0] + " files to metadata");
     update(commitMetadata, createInstantTime);
   }
 
+  /**
+   * Function to find hoodie partitions and list files in them in parallel.
+   *
+   * @param jsc
+   * @param datasetMetaClient
+   * @return Map of partition names to a list of FileStatus for all the files in the partition
+   */
+  private Map<String, List<FileStatus>> parallelFileSystemListing(JavaSparkContext jsc,
+      HoodieTableMetaClient datasetMetaClient) {
+    List<Path> pathsToList = new LinkedList<>();
+    pathsToList.add(new Path(datasetWriteConfig.getBasePath()));
+    Map<String, List<FileStatus>> partitionToFileStatus = new HashMap<>();
+
+    while (!pathsToList.isEmpty()) {
+      // List all directories in parallel
+      List<Pair<Path, FileStatus[]>> dirToFileListing =
+          jsc.parallelize(pathsToList, Math.min(pathsToList.size(), jsc.defaultParallelism()))
+            .map(path -> {
+              FileSystem fs = datasetMetaClient.getFs();

Review comment:
       Better.
   
   Except for new Configuration() I think its better to pass in the current configuration (from datasetMetaClient) so that any production configs can be carried over.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io commented on pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2343:
URL: https://github.com/apache/hudi/pull/2343#issuecomment-747754631


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=h1) Report
   > Merging [#2343](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=desc) (374bbb6) into [rfc-15](https://codecov.io/gh/apache/hudi/commit/2e09aeca47a7f14cd83f644a5132d4dfd0f27813?el=desc) (2e09aec) will **decrease** coverage by `15.55%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2343/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             rfc-15    #2343       +/-   ##
   =============================================
   - Coverage     59.44%   43.88%   -15.56%     
   + Complexity     4592     3379     -1213     
   =============================================
     Files           573      573               
     Lines         24353    24362        +9     
     Branches       2440     2440               
   =============================================
   - Hits          14476    10692     -3784     
   - Misses         8825    12654     +3829     
   + Partials       1052     1016       -36     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `27.15% <0.00%> (-39.54%)` | `0.00 <0.00> (ø)` | |
   | hudiclient | `24.34% <0.00%> (-51.02%)` | `0.00 <0.00> (ø)` | |
   | hudicommon | `52.78% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudihadoopmr | `33.05% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudispark | `67.19% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | huditimelineservice | `64.43% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudiutilities | `69.38% <ø> (ø)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2343?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YVdyaXRlci5qYXZh) | `2.16% <0.00%> (-68.76%)` | `2.00 <0.00> (-48.00)` | |
   | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Ib29kaWVHYXVnZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-24.00%)` | |
   | [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
   | [...g/apache/hudi/exception/HoodieUpsertException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZVVwc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllS2V5TG9jYXRpb25GZXRjaEhhbmRsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
   | [...apache/hudi/exception/HoodieRollbackException.java](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZVJvbGxiYWNrRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
   | ... and [137 more](https://codecov.io/gh/apache/hudi/pull/2343/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar merged pull request #2343: [HUDI-1469] Faster initialization of metadata table using parallelized listing.

Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2343:
URL: https://github.com/apache/hudi/pull/2343


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org