You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/28 01:30:53 UTC

[GitHub] [incubator-hudi] bvaradar opened a new pull request #1459: [WIP] [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

bvaradar opened a new pull request #1459: [WIP] [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459
 
 
   ## What is the purpose of the pull request
   
   This is part of code changes needed to support RFC-12 to support efficient bootstrap of legacy tables. This PR contains 2 commits
   
     * HUDI-418 - Bootstrap Index using HFile with APIs to create and lookup. Note that this implementation is different from what was initially proposed in RFC-12. This implementation uses existing HFile layout as this layout works well for our usecase and the implementation is battle tested and stable. There are 2 HFiles maintained - one for storing all bootstrap mappings at partition level (for all file-system view calls) and another at per file-id level (for compaction lookups)
      * HUDI-421 Core Changes in File System View to integrate with Bootstrap Index. With this change the upper layers which uses FileSystem View can easily identify and handle external base-files.
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hudi] bvaradar commented on pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1459:
URL: https://github.com/apache/hudi/pull/1459#issuecomment-635492831


   @umehrot2 : Will answer to the review comments in this PR during the code-review process.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] bvaradar commented on pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1459:
URL: https://github.com/apache/hudi/pull/1459#issuecomment-635492457


   @umehrot2 @vinothchandar : Closing this PR as I will be opening the master PR shortly. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403400178
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapSourceFileMapping.java
 ##########
 @@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+
+public class BootstrapSourceFileMapping implements Serializable, Comparable<BootstrapSourceFileMapping> {
+
+  private final String sourceBasePath;
+  private final String sourcePartitionPath;
+  private final String hudiPartitionPath;
+  private final HoodieFileStatus sourceFileStatus;
+  private final String hudiFileId;
+
+  public BootstrapSourceFileMapping(String sourceBasePath, String sourcePartitionPath,
+      String hudiPartitionPath, HoodieFileStatus sourceFileStatus, String hudiFileId) {
+    this.sourceBasePath = sourceBasePath;
+    this.sourcePartitionPath = sourcePartitionPath;
+    this.hudiPartitionPath = hudiPartitionPath;
+    this.sourceFileStatus = sourceFileStatus;
+    this.hudiFileId = hudiFileId;
+  }
+
+  @Override
+  public String toString() {
+    return "BootstrapSourceFileMapping{"
+        + "sourceBasePath='" + sourceBasePath + '\''
+        + ", sourcePartitionPath='" + sourcePartitionPath + '\''
+        + ", hudiPartitionPath='" + hudiPartitionPath + '\''
+        + ", sourceFileStatus='" + sourceFileStatus + '\''
+        + ", hudiFileId='" + hudiFileId + '\''
+        + '}';
 
 Review comment:
   [Suggestion] You may want to use: http://commons.apache.org/proper/commons-lang/javadocs/api-release/org/apache/commons/lang3/builder/ToStringBuilder.html

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] bvaradar commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#issuecomment-607014929
 
 
   @vinothchandar  : Ready for review (cc @umehrot2 )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403408572
 
 

 ##########
 File path: hudi-common/src/main/avro/HoodieBootstrapIndexInfo.avsc
 ##########
 @@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"BootstrapIndexInfo",
+   "fields":[
+      {
+        "name":"version",
 
 Review comment:
   I don't really see `version` being used anywhere in code, but present in all the model files. What's the reason behind it ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#issuecomment-607013294
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=h1) Report
   > Merging [#1459](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/78b3194e8241c519a85310997f31b2b55df487e1&el=desc) will **increase** coverage by `0.28%`.
   > The diff coverage is `78.27%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1459/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1459      +/-   ##
   ============================================
   + Coverage     67.64%   67.92%   +0.28%     
     Complexity      261      261              
   ============================================
     Files           348      351       +3     
     Lines         16672    17115     +443     
     Branches       1694     1726      +32     
   ============================================
   + Hits          11278    11626     +348     
   - Misses         4653     4725      +72     
   - Partials        741      764      +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `100.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `80.35% <12.50%> (-11.40%)` | `0.00 <0.00> (ø)` | |
   | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `66.33% <25.00%> (-0.84%)` | `0.00 <0.00> (ø)` | |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `82.16% <52.94%> (-6.58%)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `87.34% <57.14%> (-10.94%)` | `0.00 <0.00> (ø)` | |
   | [.../hudi/common/model/BootstrapSourceFileMapping.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0Jvb3RzdHJhcFNvdXJjZUZpbGVNYXBwaW5nLmphdmE=) | `65.51% <65.51%> (ø)` | `0.00 <0.00> (?)` | |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `67.56% <70.00%> (+0.90%)` | `0.00 <0.00> (ø)` | |
   | [...rg/apache/hudi/common/model/HoodieFileGroupId.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVHcm91cElkLmphdmE=) | `77.77% <75.00%> (-0.80%)` | `0.00 <0.00> (ø)` | |
   | [...g/apache/hudi/common/bootstrap/BootstrapIndex.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jvb3RzdHJhcC9Cb290c3RyYXBJbmRleC5qYXZh) | `80.28% <80.28%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `83.56% <81.81%> (-0.57%)` | `0.00 <0.00> (ø)` | |
   | ... and [17 more](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=footer). Last update [78b3194...434e6c3](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] bvaradar commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#issuecomment-607463279
 
 
   
   @umehrot2 : Made changes to track external file status. 
   
   @vinothchandar : Ready for review.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hudi] bvaradar closed pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
bvaradar closed pull request #1459:
URL: https://github.com/apache/hudi/pull/1459


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403362814
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
 ##########
 @@ -240,4 +244,27 @@ public static String decompress(byte[] bytes) {
       throw new HoodieIOException("IOException while decompressing text", e);
     }
   }
+
+  /**
+   * Generate a reader schema off the provided writeSchema, to just project out the provided columns.
+   */
+  public static Schema generateProjectionSchema(Schema originalSchema, List<String> fieldNames) {
+    Map<String, Field> schemaFieldsMap = originalSchema.getFields().stream()
+        .map(r -> Pair.of(r.name().toLowerCase(), r)).collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
+    List<Schema.Field> projectedFields = new ArrayList<>();
+    for (String fn : fieldNames) {
+      Schema.Field field = schemaFieldsMap.get(fn.toLowerCase());
 
 Review comment:
   I think we can avoid forming `schemaFieldsMap` here. We can directly get the field from original schema using `originalSchema.getField()` and I don't think there will be any performance impact here because Avro internally maintains a Map to similar to this to find the field instead of `O(n)`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r404463290
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java
 ##########
 @@ -104,6 +111,11 @@ protected void resetViewState() {
     return fileIdToPendingCompaction;
   }
 
+  protected Map<HoodieFileGroupId, ExternalBaseFileMapping> createFileIdToExternalBaseFileMap(
+      Map<HoodieFileGroupId, ExternalBaseFileMapping> fileGroupIdExternalBaseFileMap) {
+    return fileGroupIdExternalBaseFileMap;
+  }
 
 Review comment:
   This function doesn't do anything

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403406705
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/BootstrapIndex.java
 ##########
 @@ -0,0 +1,494 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ *  (a) At partition granularity to lookup all indices for each partition.
+ *  (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class BootstrapIndex implements Serializable, AutoCloseable {
+
+  private static final Logger LOG = LogManager.getLogger(BootstrapIndex.class);
+
+  public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+  // Used as naming extensions.
+  public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+  // Additional Metadata written to HFiles.
+  public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+  private final HoodieTableMetaClient metaClient;
+  // Base Path of external files.
+  private final String sourceBasePath;
+  // Well Known Paths for indices
+  private final String indexByPartitionPath;
+  private final String indexByFileIdPath;
+  // Flag to idenitfy if Bootstrap Index is empty or not
+  private final boolean isBootstrapped;
+
+  // Index Readers
+  private transient HFile.Reader indexByPartitionReader;
+  private transient HFile.Reader indexByFileIdReader;
+
+  // Bootstrap Index Info
+  private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+  public BootstrapIndex(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+
+    try {
+      metaClient.initializeBootstrapDirsIfNotExists();
+      Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+      Path indexByFilePath = getIndexByFileIdPath(metaClient);
+      if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+        this.indexByPartitionPath = indexByPartitionPath.toString();
+        this.indexByFileIdPath = indexByFilePath.toString();
+        this.sourceBasePath = getBootstrapIndexInfo().getSourceBasePath();
+        this.isBootstrapped = true;
+      } else {
+        this.indexByPartitionPath = null;
+        this.indexByFileIdPath = null;
+        this.sourceBasePath = null;
+        this.isBootstrapped = false;
+      }
+      LOG.info("Loaded BootstrapIndex with source base path :" + sourceBasePath);
+    } catch (IOException e) {
+      throw new HoodieIOException(e.getMessage(), e);
+    }
+  }
+
+  public BootstrapIndexInfo getBootstrapIndexInfo() {
+    if (null == bootstrapIndexInfo) {
+      synchronized (this) {
+        if (null == bootstrapIndexInfo) {
+          try {
+            bootstrapIndexInfo = fetchBootstrapIndexInfo();
+          } catch (IOException ioe) {
+            throw new HoodieException(ioe.getMessage(), ioe);
+          }
+        }
+      }
+    }
+    return bootstrapIndexInfo;
+  }
+
+  private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getBootstrapIndexByPartitionFolderName(),
+        FSUtils.makeBootstrapIndexFileName(HoodieTimeline.BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+            BOOTSTRAP_INDEX_FILE_TYPE));
+  }
+
+  private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderName(),
+        FSUtils.makeBootstrapIndexFileName(HoodieTimeline.BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+            BOOTSTRAP_INDEX_FILE_TYPE));
+  }
+
+  private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+    return TimelineMetadataUtils.deserializeAvroMetadata(
+        getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+        BootstrapIndexInfo.class);
+  }
+
+  private HFile.Reader getIndexByPartitionReader() {
+    if (null == indexByPartitionReader) {
+      synchronized (this) {
+        if (null == indexByPartitionReader) {
+          LOG.info("Opening partition index :" + indexByPartitionPath);
+          this.indexByPartitionReader =
+              createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+        }
+      }
+    }
+    return indexByPartitionReader;
+  }
+
+  private HFile.Reader getIndexByFileIdReader() {
+    if (null == indexByFileIdReader) {
+      synchronized (this) {
+        if (null == indexByFileIdReader) {
+          LOG.info("Opening fileId index :" + indexByFileIdPath);
+          this.indexByFileIdReader =
+              createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+        }
+      }
+    }
+    return indexByFileIdReader;
+  }
+
+  public List<String> getAllBootstrapPartitionKeys() {
+    HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+    return getAllKeys(scanner);
+  }
+
+  public List<String> getAllBootstrapFileIdKeys() {
+    HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+    return getAllKeys(scanner);
+  }
+
+  private List<String> getAllKeys(HFileScanner scanner) {
+    List<String> keys = new ArrayList<>();
+    if (isBootstrapped) {
+      try {
+        boolean available = scanner.seekTo();
+        while (available) {
+          keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+          available = scanner.next();
+        }
+      } catch (IOException ioe) {
+        throw new HoodieIOException(ioe.getMessage(), ioe);
+      }
+    }
+    return keys;
+  }
+
+  /**
+   * Drop Bootstrap Index.
+   */
+  public void dropIndex() {
+    try {
+      Path[] indexPaths = new Path[] { new Path(indexByPartitionPath), new Path(indexByFileIdPath) };
+      for (Path indexPath : indexPaths) {
+        if (metaClient.getFs().exists(indexPath)) {
+          LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+          metaClient.getFs().delete(indexPath);
+        }
+      }
+    } catch (IOException ioe) {
+      throw new HoodieIOException(ioe.getMessage(), ioe);
+    }
+  }
+
+  /**
+   * Lookup bootstrap index by partition.
+   * @param partition Partition to lookup
+   * @return
+   */
+  public List<BootstrapSourceFileMapping> getBootstrapInfoForPartition(String partition) {
+    if (isBootstrapped) {
+      try {
+        HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+        KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+            HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+        if (scanner.seekTo(keyValue) == 0) {
+          ByteBuffer readValue = scanner.getValue();
+          byte[] valBytes = Bytes.toBytes(readValue);
+          BootstrapPartitionMetadata metadata =
+              TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+          return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+              .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(), partition,
+                  e.getValue(), e.getKey())).collect(Collectors.toList());
+        } else {
+          LOG.info("No value found for partition key (" + partition + ")");
+          return new ArrayList<>();
+        }
+      } catch (IOException ioe) {
+        throw new HoodieIOException(ioe.getMessage(), ioe);
+      }
+    }
+    return new ArrayList<>();
+  }
+
+  public String getSourceBasePath() {
+    return sourceBasePath;
+  }
+
+  /**
+   * Lookup Bootstrap index by file group ids.
+   * @param ids File Group Ids
+   * @return
+   */
+  public Map<HoodieFileGroupId, BootstrapSourceFileMapping> getBootstrapInfoForFileIds(List<HoodieFileGroupId> ids) {
+    Map<HoodieFileGroupId, BootstrapSourceFileMapping> result = new HashMap<>();
+    // Arrange input Keys in sorted order for 1 pass scan
+    List<HoodieFileGroupId> fileGroupIds = new ArrayList<>(ids);
+    Collections.sort(fileGroupIds);
+    if (isBootstrapped) {
+      try {
+        HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+        for (HoodieFileGroupId fileGroupId : fileGroupIds) {
+          KeyValue keyValue = new KeyValue(Bytes.toBytes(getFileGroupKey(fileGroupId)), new byte[0], new byte[0],
+              HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+          if (scanner.seekTo(keyValue) == 0) {
+            ByteBuffer readValue = scanner.getValue();
+            byte[] valBytes = Bytes.toBytes(readValue);
+            BootstrapSourceFilePartitionInfo fileInfo = TimelineMetadataUtils.deserializeAvroMetadata(valBytes,
+                BootstrapSourceFilePartitionInfo.class);
+            BootstrapSourceFileMapping mapping = new BootstrapSourceFileMapping(sourceBasePath,
+                fileInfo.getSourcePartitionPath(), fileInfo.getHudiPartitionPath(), fileInfo.getSourceFileStatus(),
+                fileGroupId.getFileId());
+            result.put(fileGroupId, mapping);
+          }
+        }
+      } catch (IOException ioe) {
+        throw new HoodieIOException(ioe.getMessage(), ioe);
+      }
+    }
+    return result;
+  }
+
+  /**
+   * Helper method to create HFile Reader.
+   * @param hFilePath File Path
+   * @param conf  Configuration
+   * @param fileSystem File System
+   * @return
+   */
+  private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+    try {
+      LOG.info("Opening HFile for reading :" + hFilePath);
+      CacheConfig config = new CacheConfig(conf);
+      HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath), new CacheConfig(conf),
+          conf);
+      return reader;
 
 Review comment:
   nit: May be just do
   ```
   return HFile.createReader(fileSystem, new HFilePathForReader(hFilePath), new CacheConfig(conf), conf);
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r404458181
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
 ##########
 @@ -395,9 +448,10 @@ protected FileSlice filterBaseFileAfterPendingCompaction(FileSlice fileSlice) {
   public final Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn) {
     try {
       readLock.lock();
-      return fetchAllStoredFileGroups().map(fileGroup -> Option.fromJavaOptional(
+      return fetchAllStoredFileGroups().map(fileGroup -> Pair.of(fileGroup.getFileGroupId(), Option.fromJavaOptional(
           fileGroup.getAllBaseFiles().filter(baseFile -> commitsToReturn.contains(baseFile.getCommitTime())
-              && !isBaseFileDueToPendingCompaction(baseFile)).findFirst())).filter(Option::isPresent).map(Option::get);
+              && !isBaseFileDueToPendingCompaction(baseFile)).findFirst()))).filter(p -> p.getValue().isPresent())
+          .map(p -> addExternalBaseFileIfPresent(p.getKey(), p.getValue().get()));
 
 Review comment:
   I think it would be better if we can push the code for adding external base if present down to `fetchAllStoredFileGroups()`. It would help avoid so many changes we are making to this class to add external files, and avoid having to keep in mind to do this for future methods we might add. Also, in general I think its good if the underlying implementations return the file groups formed with all the information (including external files) instead of this class managing it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403362949
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
 ##########
 @@ -240,4 +244,27 @@ public static String decompress(byte[] bytes) {
       throw new HoodieIOException("IOException while decompressing text", e);
     }
   }
+
+  /**
+   * Generate a reader schema off the provided writeSchema, to just project out the provided columns.
+   */
+  public static Schema generateProjectionSchema(Schema originalSchema, List<String> fieldNames) {
+    Map<String, Field> schemaFieldsMap = originalSchema.getFields().stream()
+        .map(r -> Pair.of(r.name().toLowerCase(), r)).collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
+    List<Schema.Field> projectedFields = new ArrayList<>();
+    for (String fn : fieldNames) {
+      Schema.Field field = schemaFieldsMap.get(fn.toLowerCase());
 
 Review comment:
   I think we can avoid forming `schemaFieldsMap` here. We can directly get the field from original schema using `originalSchema.getField()` and I don't think there will be any performance impact here because Avro internally maintains a Map to similar to this to find the field instead of `O(n)`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r404416496
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/FileStatusUtils.java
 ##########
 @@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap;
+
+import org.apache.hudi.avro.model.HoodieFSPermission;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.avro.model.HoodiePath;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+import java.io.IOException;
+
+/**
+ * Helper functions to convert between FileStatus and HoodieFileStatus.
+ */
+public class FileStatusUtils {
+
+  public static Path toPath(HoodiePath path) {
+    if (null == path) {
+      return null;
+    }
+    return new Path(path.getUri());
+  }
+
+  public static HoodiePath fromPath(Path path) {
+    if (null == path) {
+      return null;
+    }
+    return HoodiePath.newBuilder().setUri(path.toUri().toString()).build();
+  }
+  
+  public static FsPermission toFSPermission(HoodieFSPermission fsPermission) {
+    if (null == fsPermission) {
+      return null;
+    }
+    FsAction userAction = fsPermission.getUserAction() != null ? FsAction.valueOf(fsPermission.getUserAction()) : null;
+    FsAction grpAction = fsPermission.getGroupAction() != null ? FsAction.valueOf(fsPermission.getGroupAction()) : null;
+    FsAction otherAction =
+        fsPermission.getOtherAction() != null ? FsAction.valueOf(fsPermission.getOtherAction()) : null;
+    boolean stickyBit = fsPermission.getStickyBit() != null ? fsPermission.getStickyBit() : false;
+    return new FsPermission(userAction, grpAction, otherAction, stickyBit);
+  }
+
+  public static HoodieFSPermission fromFSPermission(FsPermission fsPermission) {
+    if (null == fsPermission) {
+      return null;
+    }
+    String userAction = fsPermission.getUserAction() != null ? fsPermission.getUserAction().name() : null;
+    String grpAction = fsPermission.getGroupAction() != null ? fsPermission.getGroupAction().name() : null;
+    String otherAction = fsPermission.getOtherAction() != null ? fsPermission.getOtherAction().name() : null;
+    return HoodieFSPermission.newBuilder().setUserAction(userAction).setGroupAction(grpAction)
+        .setOtherAction(otherAction).setStickyBit(fsPermission.getStickyBit()).build();
+  }
+  
+  public static FileStatus toFileStatus(HoodieFileStatus fileStatus) {
+    if (null == fileStatus) {
+      return null;
+    }
+
+    System.out.println("HoodieFileStatus : " + fileStatus);
 
 Review comment:
   nit: remove

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403362814
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
 ##########
 @@ -240,4 +244,27 @@ public static String decompress(byte[] bytes) {
       throw new HoodieIOException("IOException while decompressing text", e);
     }
   }
+
+  /**
+   * Generate a reader schema off the provided writeSchema, to just project out the provided columns.
+   */
+  public static Schema generateProjectionSchema(Schema originalSchema, List<String> fieldNames) {
+    Map<String, Field> schemaFieldsMap = originalSchema.getFields().stream()
+        .map(r -> Pair.of(r.name().toLowerCase(), r)).collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
+    List<Schema.Field> projectedFields = new ArrayList<>();
+    for (String fn : fieldNames) {
+      Schema.Field field = schemaFieldsMap.get(fn.toLowerCase());
 
 Review comment:
   I think we can avoid forming `schemaFieldsMap` here. We can directly get the field from original schema using `originalSchema.getField()` and I don't think there will be any performance impact here because Avro internally maintains a Map to similar to this to find the field instead of `O(n)`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r404455728
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
 ##########
 @@ -328,7 +377,8 @@ protected FileSlice filterBaseFileAfterPendingCompaction(FileSlice fileSlice) {
       readLock.lock();
       String partitionPath = formatPartitionKey(partitionStr);
       ensurePartitionLoadedCorrectly(partitionPath);
-      return fetchLatestBaseFiles(partitionPath);
+      return fetchLatestBaseFiles(partitionPath)
+          .map(df -> addExternalBaseFileIfPresent(new HoodieFileGroupId(partitionPath, df.getFileId()), df));
 
 Review comment:
   Do we need to add external file again, given `fetchLatestBaseFiles` already adds the external file ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on issue #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#issuecomment-607572115
 
 
   > @umehrot2 : Made changes to track external file status.
   
   Thanks ! Plan to start reviewing this tomorrow.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r404418022
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/FileStatusUtils.java
 ##########
 @@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap;
+
+import org.apache.hudi.avro.model.HoodieFSPermission;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.avro.model.HoodiePath;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+import java.io.IOException;
+
+/**
+ * Helper functions to convert between FileStatus and HoodieFileStatus.
+ */
+public class FileStatusUtils {
+
+  public static Path toPath(HoodiePath path) {
+    if (null == path) {
+      return null;
+    }
+    return new Path(path.getUri());
+  }
+
+  public static HoodiePath fromPath(Path path) {
+    if (null == path) {
+      return null;
+    }
+    return HoodiePath.newBuilder().setUri(path.toUri().toString()).build();
+  }
+  
+  public static FsPermission toFSPermission(HoodieFSPermission fsPermission) {
+    if (null == fsPermission) {
+      return null;
+    }
+    FsAction userAction = fsPermission.getUserAction() != null ? FsAction.valueOf(fsPermission.getUserAction()) : null;
+    FsAction grpAction = fsPermission.getGroupAction() != null ? FsAction.valueOf(fsPermission.getGroupAction()) : null;
+    FsAction otherAction =
+        fsPermission.getOtherAction() != null ? FsAction.valueOf(fsPermission.getOtherAction()) : null;
+    boolean stickyBit = fsPermission.getStickyBit() != null ? fsPermission.getStickyBit() : false;
+    return new FsPermission(userAction, grpAction, otherAction, stickyBit);
+  }
+
+  public static HoodieFSPermission fromFSPermission(FsPermission fsPermission) {
+    if (null == fsPermission) {
+      return null;
+    }
+    String userAction = fsPermission.getUserAction() != null ? fsPermission.getUserAction().name() : null;
+    String grpAction = fsPermission.getGroupAction() != null ? fsPermission.getGroupAction().name() : null;
+    String otherAction = fsPermission.getOtherAction() != null ? fsPermission.getOtherAction().name() : null;
+    return HoodieFSPermission.newBuilder().setUserAction(userAction).setGroupAction(grpAction)
+        .setOtherAction(otherAction).setStickyBit(fsPermission.getStickyBit()).build();
+  }
+  
+  public static FileStatus toFileStatus(HoodieFileStatus fileStatus) {
+    if (null == fileStatus) {
+      return null;
+    }
+
+    System.out.println("HoodieFileStatus : " + fileStatus);
+    return new FileStatus(fileStatus.getLength(), fileStatus.getIsDir() == null ? false : fileStatus.getIsDir(),
+        fileStatus.getBlockReplication(), fileStatus.getBlockSize(), fileStatus.getModificationTime(),
+        fileStatus.getAccessTime(), toFSPermission(fileStatus.getPermission()), fileStatus.getOwner(),
+        fileStatus.getGroup(), toPath(fileStatus.getSymlink()), toPath(fileStatus.getPath()));
+  }
+
+  public static HoodieFileStatus fromFileStatus(FileStatus fileStatus) {
+    if (null == fileStatus) {
+      return null;
+    }
+
+    HoodieFileStatus fStatus = new HoodieFileStatus();
+    try {
+      fStatus.setPath(fromPath(fileStatus.getPath()));
+      fStatus.setLength(fileStatus.getLen());
+      fStatus.setIsDir(fileStatus.isDirectory());
+      fStatus.setBlockReplication((int) fileStatus.getReplication());
+      fStatus.setBlockSize(fileStatus.getBlockSize());
+      fStatus.setModificationTime(fileStatus.getModificationTime());
+      fStatus.setAccessTime(fileStatus.getModificationTime());
+      fStatus.setSymlink(fileStatus.isSymlink() ? fromPath(fileStatus.getSymlink()) : null);
+      safeReadAndSetMetadata(fStatus, fileStatus);
+    } catch (IOException ioe) {
+      throw new HoodieIOException(ioe.getMessage(), ioe);
+    }
+    return fStatus;
+  }
+
+  /**
+   * Used to safely handle FileStatus calls which might fail on some FileSystem implementation.
+   * (DeprecatedLocalFileSystem)
+   */
+  private static void safeReadAndSetMetadata(HoodieFileStatus fStatus, FileStatus fileStatus) {
+    try {
+      fStatus.setOwner(fileStatus.getOwner());
+      fStatus.setGroup(fileStatus.getGroup());
+      fStatus.setPermission(fromFSPermission(fileStatus.getPermission()));
+    } catch (IllegalArgumentException ie) {
+      // Deprecated File System (testing) does not work well with this call
 
 Review comment:
   May be log a warning ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403391721
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
 ##########
 @@ -375,6 +395,7 @@ public static HoodieTableMetaClient initTableAndGetMetaClient(Configuration hado
       fs.mkdirs(auxiliaryFolder);
     }
 
+    initializeBootstrapDirsIfNotExists(hadoopConf, basePath, fs);
 
 Review comment:
   Seems like this can be an optional depending on whether or not the user chooses to bootstrap. Its okay if that work is going to be part of a separate PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403394637
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
 ##########
 @@ -111,6 +112,10 @@ public static String makeDataFileName(String instantTime, String writeToken, Str
     return String.format("%s_%s_%s.parquet", fileId, writeToken, instantTime);
   }
 
+  public static String makeBootstrapIndexFileName(String instantTime, String fileId, String fileType) {
+    return String.format("%s_%s_%s.%s", fileId, "1-0-1", instantTime, fileType);
 
 Review comment:
   Just for my understanding is there is a reasoning behind `1-0-1` here ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io commented on issue #1459: [WIP] [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1459: [WIP] [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#issuecomment-607013294
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=h1) Report
   > Merging [#1459](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/78b3194e8241c519a85310997f31b2b55df487e1&el=desc) will **increase** coverage by `0.28%`.
   > The diff coverage is `78.27%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1459/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1459      +/-   ##
   ============================================
   + Coverage     67.64%   67.92%   +0.28%     
     Complexity      261      261              
   ============================================
     Files           348      351       +3     
     Lines         16672    17115     +443     
     Branches       1694     1726      +32     
   ============================================
   + Hits          11278    11626     +348     
   - Misses         4653     4725      +72     
   - Partials        741      764      +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `100.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `80.35% <12.50%> (-11.40%)` | `0.00 <0.00> (ø)` | |
   | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `66.33% <25.00%> (-0.84%)` | `0.00 <0.00> (ø)` | |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `82.16% <52.94%> (-6.58%)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `87.34% <57.14%> (-10.94%)` | `0.00 <0.00> (ø)` | |
   | [.../hudi/common/model/BootstrapSourceFileMapping.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0Jvb3RzdHJhcFNvdXJjZUZpbGVNYXBwaW5nLmphdmE=) | `65.51% <65.51%> (ø)` | `0.00 <0.00> (?)` | |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `67.56% <70.00%> (+0.90%)` | `0.00 <0.00> (ø)` | |
   | [...rg/apache/hudi/common/model/HoodieFileGroupId.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVHcm91cElkLmphdmE=) | `77.77% <75.00%> (-0.80%)` | `0.00 <0.00> (ø)` | |
   | [...g/apache/hudi/common/bootstrap/BootstrapIndex.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jvb3RzdHJhcC9Cb290c3RyYXBJbmRleC5qYXZh) | `80.28% <80.28%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `83.56% <81.81%> (-0.57%)` | `0.00 <0.00> (ø)` | |
   | ... and [17 more](https://codecov.io/gh/apache/incubator-hudi/pull/1459/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=footer). Last update [78b3194...434e6c3](https://codecov.io/gh/apache/incubator-hudi/pull/1459?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test

Posted by GitBox <gi...@apache.org>.
umehrot2 commented on a change in pull request #1459: [HUDI-418] [HUDI-421] Bootstrap Index using HFile and File System View Changes with unit-test
URL: https://github.com/apache/incubator-hudi/pull/1459#discussion_r403394224
 
 

 ##########
 File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
 ##########
 @@ -207,6 +213,20 @@ public String getMetaAuxiliaryPath() {
     return basePath + File.separator + AUXILIARYFOLDER_NAME;
   }
 
+  /**
+   * @return Bootstrap Index By Partition Folder
+   */
+  public String getBootstrapIndexByPartitionFolderName() {
+    return getMetaAuxiliaryPath() + File.separator + BOOTSTRAP_INDEX_BY_PARTITION_FOLDER_NAME;
+  }
+
+  /**
+   * @return Bootstrap Index By Hudi File Id Folder
+   */
+  public String getBootstrapIndexByFileIdFolderNameFolderName() {
+    return getMetaAuxiliaryPath() + File.separator + BOOTSTRAP_INDEX_BY_FILE_ID_FOLDER_NAME;
+  }
 
 Review comment:
   `BOOTSTRAP_INDEX_BY_PARTITION_FOLDER_NAME` and `BOOTSTRAP_INDEX_BY_FILE_ID_FOLDER_NAME` both already contain `AUXILIARYFOLDER_NAME` in it. Here we are again prefixing the auxiliary path while returning. I think this would be prefixing the path twice with aux folder. Seems like a bug.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services