You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/14 06:24:13 UTC

[GitHub] [incubator-hudi] xushiyan opened a new pull request #1405: [HUDI-344] Add partitioner param to Exporter

xushiyan opened a new pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405
 
 
   * To allow custom repartitioning during export
   
   ## Verify this pull request
   
   This pull request is already covered by existing tests, such as `org.apache.hudi.utilities.TestHoodieSnapshotExporter`
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/e3019031d8fff60df4fec82eac3fd5c044011635&el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `95.65%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.70%   67.75%   +0.04%     
   - Complexity      243      245       +2     
   ============================================
     Files           338      338              
     Lines         16374    16377       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11086    11096      +10     
   + Misses         4548     4543       -5     
   + Partials        740      738       -2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22.00 <4.00> (+2.00)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0.00%> (+5.55%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [e301903...7b00b57](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `86.95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.69%   67.71%   +0.01%     
   - Complexity      243      244       +1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11088       +5     
   + Misses         4548     4546       -2     
     Partials        740      740
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21 <3> (+1)` | :arrow_up: |
   | [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `25% <0%> (-50%)` | `0% <0%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf merged pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] xushiyan commented on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599247813
 
 
   @leesf Addressed the comments in the last commit. Thank you!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/e3019031d8fff60df4fec82eac3fd5c044011635?src=pr&el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `95.65%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage      67.7%   67.75%   +0.04%     
   - Complexity      243      245       +2     
   ============================================
     Files           338      338              
     Lines         16374    16377       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11086    11096      +10     
   + Misses         4548     4543       -5     
   + Partials        740      738       -2
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22 <4> (+2)` | :arrow_up: |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0%> (+5.55%)` | `0% <0%> (ø)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [e301903...7b00b57](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `95.65%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.69%   67.74%   +0.04%     
   - Complexity      243      245       +2     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11092       +9     
   + Misses         4548     4543       -5     
   + Partials        740      739       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22.00 <4.00> (+2.00)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...4ffb4ba](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r393533586
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -45,41 +50,65 @@
 import org.apache.spark.sql.Row;
 import org.apache.spark.sql.SaveMode;
 import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.execution.datasources.DataSource;
-
-import scala.Tuple2;
-import scala.collection.JavaConversions;
 
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.stream.Collectors;
 
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
 /**
  * Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
  *
  * @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
  */
 public class HoodieSnapshotExporter {
+
+  @FunctionalInterface
+  public interface Partitioner {
+
+    DataFrameWriter<Row> partition(Dataset<Row> source);
+
+  }
+
   private static final Logger LOG = LogManager.getLogger(HoodieSnapshotExporter.class);
 
+  public static class OutputFormatValidator implements IValueValidator<String> {
+
+    static final String HUDI = "hudi";
+    static final List<String> FORMATS = ImmutableList.of("json", "parquet", HUDI);
 
 Review comment:
   should we document this that only support three formats?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392595615
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -113,21 +126,7 @@ public int export(SparkSession spark, Config cfg) throws IOException {
         return -1;
       }
       if (!cfg.outputFormat.equalsIgnoreCase("hudi")) {
 
 Review comment:
   "hudi" would be constant?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io commented on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **decrease** coverage by `<.01%`.
   > The diff coverage is `60.86%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   - Coverage     67.69%   67.69%   -0.01%     
   - Complexity      243      244       +1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11084       +1     
   - Misses         4548     4550       +2     
     Partials        740      740
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `75.72% <60.86%> (-1.28%)` | `21 <3> (+1)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...c6cbfca](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `86.95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.69%   67.73%   +0.03%     
   + Complexity      243      242       -1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11091       +8     
   + Misses         4548     4544       -4     
   + Partials        740      739       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21.00 <3.00> (+1.00)` | |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0.00%> (+5.55%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `86.95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.69%   67.73%   +0.03%     
   + Complexity      243      242       -1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11091       +8     
   + Misses         4548     4544       -4     
   + Partials        740      739       -1
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21 <3> (+1)` | :arrow_up: |
   | [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0%> (+5.55%)` | `0% <0%> (ø)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392606703
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -113,21 +126,7 @@ public int export(SparkSession spark, Config cfg) throws IOException {
         return -1;
       }
       if (!cfg.outputFormat.equalsIgnoreCase("hudi")) {
 
 Review comment:
   Sure. Will fix this once #1404 merged, with explicit format checking.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r393872943
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -45,41 +50,65 @@
 import org.apache.spark.sql.Row;
 import org.apache.spark.sql.SaveMode;
 import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.execution.datasources.DataSource;
-
-import scala.Tuple2;
-import scala.collection.JavaConversions;
 
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.stream.Collectors;
 
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
 /**
  * Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
  *
  * @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
  */
 public class HoodieSnapshotExporter {
+
+  @FunctionalInterface
+  public interface Partitioner {
+
+    DataFrameWriter<Row> partition(Dataset<Row> source);
+
+  }
+
   private static final Logger LOG = LogManager.getLogger(HoodieSnapshotExporter.class);
 
+  public static class OutputFormatValidator implements IValueValidator<String> {
+
+    static final String HUDI = "hudi";
+    static final List<String> FORMATS = ImmutableList.of("json", "parquet", HUDI);
 
 Review comment:
   @leesf Sure add some explanation in the CLI help line

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **decrease** coverage by `<.01%`.
   > The diff coverage is `60.86%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&token=VTTXabwbs2&height=150&src=pr)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   - Coverage     67.69%   67.69%   -0.01%     
   - Complexity      243      244       +1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11084       +1     
   - Misses         4548     4550       +2     
     Partials        740      740
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `75.72% <60.86%> (-1.28%)` | `21 <3> (+1)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...c6cbfca](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392595777
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -47,36 +49,47 @@
 import org.apache.spark.sql.SparkSession;
 import org.apache.spark.sql.execution.datasources.DataSource;
 
-import scala.Tuple2;
-import scala.collection.JavaConversions;
-
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.stream.Collectors;
 
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
 /**
  * Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
  *
  * @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
  */
 public class HoodieSnapshotExporter {
+
+  @FunctionalInterface
+  public interface Partitioner {
+
+    DataFrameWriter<Row> partition(Dataset<Row> source);
+
+  }
 
 Review comment:
   Also need UT for user-defined Partitioner?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
   > Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `86.95%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-hudi/pull/1405/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1405      +/-   ##
   ============================================
   + Coverage     67.69%   67.71%   +0.01%     
   - Complexity      243      244       +1     
   ============================================
     Files           338      338              
     Lines         16371    16374       +3     
     Branches       1672     1674       +2     
   ============================================
   + Hits          11083    11088       +5     
   + Misses         4548     4546       -2     
     Partials        740      740              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21.00 <3.00> (+1.00)` | |
   | [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `25.00% <0.00%> (-50.00%)` | `0.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392606711
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
 ##########
 @@ -47,36 +49,47 @@
 import org.apache.spark.sql.SparkSession;
 import org.apache.spark.sql.execution.datasources.DataSource;
 
-import scala.Tuple2;
-import scala.collection.JavaConversions;
-
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.stream.Collectors;
 
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
 /**
  * Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
  *
  * @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
  */
 public class HoodieSnapshotExporter {
+
+  @FunctionalInterface
+  public interface Partitioner {
+
+    DataFrameWriter<Row> partition(Dataset<Row> source);
+
+  }
 
 Review comment:
   Sure. Will fix this once #1404 merged.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services