You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/16 02:41:13 UTC

[GitHub] [incubator-hudi] wangxianghu opened a new pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

wangxianghu opened a new pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409
 
 
   ## *Tips*
   - *Thank you very much for contributing to Apache Hudi.*
   - *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
   
   ## What is the purpose of the pull request
   
   *Add javadoc and comments to hudi write method link*
   
   ## Brief change log
   
   *Add javadoc and comments to hudi write method link*
     - *Add javadoc*
     - *Add some comments in methods*
     - *Code clean*
   
   ## Verify this pull request
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046995
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -218,6 +218,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records, whose location (file group/file id) mapped by index exists.
 
 Review comment:
   Done, PTAL

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r397124853
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -218,6 +218,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records, whose location (file group/file id) mapped by index exists.
 
 Review comment:
   Hi  @nsivabalan. I have addressed all of your comments, please take another look

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046670
 
 

 ##########
 File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
 ##########
 @@ -110,9 +110,9 @@ private void setupDemo() throws Exception {
     executeCommandStringsInDocker(PRESTO_COORDINATOR, cmds);
 
     // copy presto sql files to presto coordinator
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH1_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH2_RELATIVE_PATH, HDFS_DATA_DIR);
+    executePrestoCopyCommand(System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
 
 Review comment:
   Not really, it is still the checkstyle from incubator-hudi/style/checkstyle.xml. When I finish adding the doc and comments,I am not sure whether the changes I made can meet up the checkstyle requirements, so I scanned the entire project using the checkstyle plugin and found these checkstyle issues, so I 'corrected' them . Are theses changes in checkstyle right? should i revert them?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io commented on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.00%`.
   > The diff coverage is `83.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #1409   +/-   ##
   =========================================
     Coverage     67.69%   67.70%           
     Complexity      243      243           
   =========================================
     Files           338      338           
     Lines         16371    16372    +1     
     Branches       1672     1670    -2     
   =========================================
   + Hits          11083    11084    +1     
     Misses         4548     4548           
     Partials        740      740           
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `70.89% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `76.77% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `50.56% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `79.79% <ø> (ø)` | `8.00 <0.00> (ø)` | |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.54% <75.00%> (+0.13%)` | `38.00 <0.00> (ø)` | |
   | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `84.84% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [23afe7a...0c957c4](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396041291
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -227,7 +234,7 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
           .filter(r -> !((HoodieRecord<HoodieRecordPayload>) r).isCurrentLocationKnown());
     } catch (TableNotFoundException e) {
       // this will be executed when there is no hoodie table yet
-      // so no dups to drop
+      // so no dupes to drop
 
 Review comment:
   minor: duplicates

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046670
 
 

 ##########
 File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
 ##########
 @@ -110,9 +110,9 @@ private void setupDemo() throws Exception {
     executeCommandStringsInDocker(PRESTO_COORDINATOR, cmds);
 
     // copy presto sql files to presto coordinator
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH1_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH2_RELATIVE_PATH, HDFS_DATA_DIR);
+    executePrestoCopyCommand(System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
 
 Review comment:
   @nsivabalan, not really, it is still the check style from incubator-hudi/style/checkstyle.xml. When I finish adding the doc and comments,I am not sure whether the changes I made meet up the checkstyle requirements, so I scanned the entire project using the checkstyle plugin and found these checkstyle issues, so I 'corrected' them. Are these changes in checkstyle right? should i revert them?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046995
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -218,6 +218,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records, whose location (file group/file id) mapped by index exists.
 
 Review comment:
   @nsivabalan  Done, PTAL

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396041376
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -218,6 +218,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records, whose location (file group/file id) mapped by index exists.
 
 Review comment:
   we could make this more descriptive. "Drop duplicate records from incoming records. Drop those records that are already present in the dataset"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **increase** coverage by `<.01%`.
   > The diff coverage is `83.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##             master   #1409      +/-   ##
   ===========================================
   + Coverage     67.69%   67.7%   +<.01%     
     Complexity      243     243              
   ===========================================
     Files           338     338              
     Lines         16371   16372       +1     
     Branches       1672    1670       -2     
   ===========================================
   + Hits          11083   11084       +1     
     Misses         4548    4548              
     Partials        740     740
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `50.56% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `79.79% <ø> (ø)` | `8 <0> (ø)` | :arrow_down: |
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `70.89% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `76.77% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `84.84% <100%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.54% <75%> (+0.13%)` | `38 <0> (ø)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [23afe7a...0c957c4](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/c5030f77a0e63f609ed2c674bea00201b97d8bb6&el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `84.61%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1409      +/-   ##
   ============================================
   + Coverage     67.60%   67.61%   +0.01%     
   + Complexity      255      254       -1     
   ============================================
     Files           340      340              
     Lines         16514    16514              
     Branches       1689     1687       -2     
   ============================================
   + Hits          11164    11166       +2     
   + Misses         4612     4610       -2     
     Partials        738      738              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `70.89% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `76.77% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `50.56% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `80.20% <ø> (ø)` | `8.00 <0.00> (ø)` | |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.41% <77.77%> (+0.13%)` | `38.00 <0.00> (ø)` | |
   | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `84.84% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.50% <0.00%> (-0.74%)` | `5.00% <0.00%> (-1.00%)` | |
   | ... and [1 more](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [c5030f7...dd443bb](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046670
 
 

 ##########
 File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
 ##########
 @@ -110,9 +110,9 @@ private void setupDemo() throws Exception {
     executeCommandStringsInDocker(PRESTO_COORDINATOR, cmds);
 
     // copy presto sql files to presto coordinator
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH1_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH2_RELATIVE_PATH, HDFS_DATA_DIR);
+    executePrestoCopyCommand(System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
 
 Review comment:
   Not really, it is still the checkstyle from incubator-hudi/style/checkstyle.xml. When I finish adding the doc and comments,I am not sure whether the changes I made meet up the checkstyle requirements, so I scanned the entire project using the checkstyle plugin and found these checkstyle issues, so I 'corrected' them. Are these changes in checkstyle right? should i revert them?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046725
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -227,7 +234,7 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
           .filter(r -> !((HoodieRecord<HoodieRecordPayload>) r).isCurrentLocationKnown());
     } catch (TableNotFoundException e) {
       // this will be executed when there is no hoodie table yet
-      // so no dups to drop
+      // so no dupes to drop
 
 Review comment:
   got it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-601280309
 
 
   @nsivabalan could you please review this

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046670
 
 

 ##########
 File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
 ##########
 @@ -110,9 +110,9 @@ private void setupDemo() throws Exception {
     executeCommandStringsInDocker(PRESTO_COORDINATOR, cmds);
 
     // copy presto sql files to presto coordinator
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH1_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH2_RELATIVE_PATH, HDFS_DATA_DIR);
+    executePrestoCopyCommand(System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
 
 Review comment:
   Not really, it is still the checkstyle from incubator-hudi/style/checkstyle.xml. When I finish adding the doc and comments,I am not sure whether the changes I made meet up the checkstyle requirements, so I scanned the entire project using the checkstyle plugin and found these checkstyle issues, so I 'corrected' them . Are theses changes in checkstyle right? should i revert them?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396046725
 
 

 ##########
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##########
 @@ -227,7 +234,7 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
           .filter(r -> !((HoodieRecord<HoodieRecordPayload>) r).isCurrentLocationKnown());
     } catch (TableNotFoundException e) {
       // this will be executed when there is no hoodie table yet
-      // so no dups to drop
+      // so no dupes to drop
 
 Review comment:
   @nsivabalan  got it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r396041231
 
 

 ##########
 File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
 ##########
 @@ -110,9 +110,9 @@ private void setupDemo() throws Exception {
     executeCommandStringsInDocker(PRESTO_COORDINATOR, cmds);
 
     // copy presto sql files to presto coordinator
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH1_RELATIVE_PATH, HDFS_DATA_DIR);
-    executePrestoCopyCommand( System.getProperty("user.dir") + "/.." + PRESTO_INPUT_BATCH2_RELATIVE_PATH, HDFS_DATA_DIR);
+    executePrestoCopyCommand(System.getProperty("user.dir") + "/.." + PRESTO_INPUT_TABLE_CHECK_RELATIVE_PATH, HDFS_DATA_DIR);
 
 Review comment:
   may I know why we see just check style changes in few files? was there any change in check style recently ? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1409: [HUDI-714]Add javadoc and comments to hudi write method link
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/c5030f77a0e63f609ed2c674bea00201b97d8bb6?src=pr&el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `84.61%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1409      +/-   ##
   ============================================
   + Coverage      67.6%   67.61%   +0.01%     
   + Complexity      255      254       -1     
   ============================================
     Files           340      340              
     Lines         16514    16514              
     Branches       1689     1687       -2     
   ============================================
   + Hits          11164    11166       +2     
   + Misses         4612     4610       -2     
     Partials        738      738
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `50.56% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `80.2% <ø> (ø)` | `8 <0> (ø)` | :arrow_down: |
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `70.89% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `76.77% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `84.84% <100%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.41% <77.77%> (+0.13%)` | `38 <0> (ø)` | :arrow_down: |
   | [...java/org/apache/hudi/utilities/sources/Source.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvU291cmNlLmphdmE=) | `87.5% <0%> (-0.74%)` | `5% <0%> (-1%)` | |
   | ... and [1 more](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [c5030f7...dd443bb](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-628765699


   @nsivabalan can you please re-review and see this home


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] nsivabalan commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r425540622



##########
File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -241,6 +241,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records from incoming records.

Review comment:
       actually, my bad. we should word it differently. "Drop records already present in the dataset". Drop duplicates might give a wrong notion. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] wangxianghu commented on a change in pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409#discussion_r425775385



##########
File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -241,6 +241,13 @@ public static HoodieRecord createHoodieRecord(GenericRecord gr, Comparable order
     return new HoodieRecord<>(hKey, payload);
   }
 
+  /**
+   * Drop duplicate records from incoming records.

Review comment:
       @nsivabalan Done, thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] nsivabalan merged pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] codecov-io edited a comment on pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873


   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/9059bce977cee98e2d65769622c46a1941c563dd&el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `75.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1409      +/-   ##
   ============================================
   + Coverage     71.81%   71.85%   +0.03%     
     Complexity      294      294              
   ============================================
     Files           385      385              
     Lines         16540    16541       +1     
     Branches       1661     1659       -2     
   ============================================
   + Hits          11878    11885       +7     
   + Misses         3932     3925       -7     
   - Partials        730      731       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `77.35% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `83.11% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `56.70% <ø> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `80.00% <0.00%> (ø)` | `11.00 <0.00> (ø)` | |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.58% <75.00%> (+0.13%)` | `37.00 <0.00> (ø)` | |
   | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <100.00%> (ø)` | `18.00 <0.00> (ø)` | |
   | [...src/main/java/org/apache/hudi/metrics/Metrics.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzLmphdmE=) | `67.56% <0.00%> (+10.81%)` | `0.00% <0.00%> (ø%)` | |
   | [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `80.00% <0.00%> (+40.00%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [9059bce...1c07619](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-hudi] codecov-io edited a comment on pull request #1409: [HUDI-714]Add javadoc and comments to hudi write method link

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #1409:
URL: https://github.com/apache/incubator-hudi/pull/1409#issuecomment-599323873


   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=h1) Report
   > Merging [#1409](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/9059bce977cee98e2d65769622c46a1941c563dd&el=desc) will **decrease** coverage by `0.02%`.
   > The diff coverage is `75.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1409/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1409      +/-   ##
   ============================================
   - Coverage     71.81%   71.78%   -0.03%     
   - Complexity      294     1090     +796     
   ============================================
     Files           385      385              
     Lines         16540    16600      +60     
     Branches       1661     1666       +5     
   ============================================
   + Hits          11878    11917      +39     
   - Misses         3932     3955      +23     
   + Partials        730      728       -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `77.35% <ø> (ø)` | `13.00 <0.00> (+13.00)` | |
   | [...in/java/org/apache/hudi/table/WorkloadProfile.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvV29ya2xvYWRQcm9maWxlLmphdmE=) | `100.00% <ø> (ø)` | `9.00 <0.00> (+9.00)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `88.88% <ø> (ø)` | `4.00 <0.00> (+4.00)` | |
   | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `83.11% <ø> (ø)` | `30.00 <0.00> (+30.00)` | |
   | [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `55.55% <ø> (-1.15%)` | `0.00 <0.00> (ø)` | |
   | [...i/utilities/deltastreamer/HoodieDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllRGVsdGFTdHJlYW1lci5qYXZh) | `80.00% <0.00%> (ø)` | `11.00 <0.00> (ø)` | |
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.58% <75.00%> (+0.13%)` | `37.00 <0.00> (ø)` | |
   | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <100.00%> (ø)` | `18.00 <0.00> (ø)` | |
   | [...le/action/rollback/BaseRollbackActionExecutor.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL3JvbGxiYWNrL0Jhc2VSb2xsYmFja0FjdGlvbkV4ZWN1dG9yLmphdmE=) | `70.83% <0.00%> (-6.95%)` | `0.00% <0.00%> (ø%)` | |
   | [.../apache/hudi/common/table/TableSchemaResolver.java](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `56.71% <0.00%> (-5.47%)` | `0.00% <0.00%> (ø%)` | |
   | ... and [27 more](https://codecov.io/gh/apache/incubator-hudi/pull/1409/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=footer). Last update [9059bce...406c3e9](https://codecov.io/gh/apache/incubator-hudi/pull/1409?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org