You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/14 06:24:13 UTC
[GitHub] [incubator-hudi] xushiyan opened a new pull request #1405:
[HUDI-344] Add partitioner param to Exporter
xushiyan opened a new pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405
* To allow custom repartitioning during export
## Verify this pull request
This pull request is already covered by existing tests, such as `org.apache.hudi.utilities.TestHoodieSnapshotExporter`
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/e3019031d8fff60df4fec82eac3fd5c044011635&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `95.65%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.70% 67.75% +0.04%
- Complexity 243 245 +2
============================================
Files 338 338
Lines 16374 16377 +3
Branches 1672 1674 +2
============================================
+ Hits 11086 11096 +10
+ Misses 4548 4543 -5
+ Partials 740 738 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22.00 <4.00> (+2.00)` | |
| [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0.00%> (+5.55%)` | `0.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [e301903...7b00b57](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `86.95%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.69% 67.71% +0.01%
- Complexity 243 244 +1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11088 +5
+ Misses 4548 4546 -2
Partials 740 740
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21 <3> (+1)` | :arrow_up: |
| [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `25% <0%> (-50%)` | `0% <0%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf merged pull request #1405: [HUDI-344] Add
partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] xushiyan commented on issue #1405: [HUDI-344] Add
partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599247813
@leesf Addressed the comments in the last commit. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/e3019031d8fff60df4fec82eac3fd5c044011635?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `95.65%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.7% 67.75% +0.04%
- Complexity 243 245 +2
============================================
Files 338 338
Lines 16374 16377 +3
Branches 1672 1674 +2
============================================
+ Hits 11086 11096 +10
+ Misses 4548 4543 -5
+ Partials 740 738 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22 <4> (+2)` | :arrow_up: |
| [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0%> (+5.55%)` | `0% <0%> (ø)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [e301903...7b00b57](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `95.65%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.69% 67.74% +0.04%
- Complexity 243 245 +2
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11092 +9
+ Misses 4548 4543 -5
+ Partials 740 739 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `83.49% <95.65%> (+6.49%)` | `22.00 <4.00> (+2.00)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...4ffb4ba](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r393533586
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -45,41 +50,65 @@
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.execution.datasources.DataSource;
-
-import scala.Tuple2;
-import scala.collection.JavaConversions;
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
/**
* Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
*
* @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
*/
public class HoodieSnapshotExporter {
+
+ @FunctionalInterface
+ public interface Partitioner {
+
+ DataFrameWriter<Row> partition(Dataset<Row> source);
+
+ }
+
private static final Logger LOG = LogManager.getLogger(HoodieSnapshotExporter.class);
+ public static class OutputFormatValidator implements IValueValidator<String> {
+
+ static final String HUDI = "hudi";
+ static final List<String> FORMATS = ImmutableList.of("json", "parquet", HUDI);
Review comment:
should we document this that only support three formats?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392595615
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -113,21 +126,7 @@ public int export(SparkSession spark, Config cfg) throws IOException {
return -1;
}
if (!cfg.outputFormat.equalsIgnoreCase("hudi")) {
Review comment:
"hudi" would be constant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io commented on issue #1405: [HUDI-344]
Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `60.86%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
- Coverage 67.69% 67.69% -0.01%
- Complexity 243 244 +1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11084 +1
- Misses 4548 4550 +2
Partials 740 740
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `75.72% <60.86%> (-1.28%)` | `21 <3> (+1)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...c6cbfca](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `86.95%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.69% 67.73% +0.03%
+ Complexity 243 242 -1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11091 +8
+ Misses 4548 4544 -4
+ Partials 740 739 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21.00 <3.00> (+1.00)` | |
| [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0.00%> (+5.55%)` | `0.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `86.95%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.69% 67.73% +0.03%
+ Complexity 243 242 -1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11091 +8
+ Misses 4548 4544 -4
+ Partials 740 739 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21 <3> (+1)` | :arrow_up: |
| [...n/java/org/apache/hudi/common/model/HoodieKey.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUtleS5qYXZh) | `94.44% <0%> (+5.55%)` | `0% <0%> (ø)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] xushiyan commented on a change in pull request
#1405: [HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392606703
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -113,21 +126,7 @@ public int export(SparkSession spark, Config cfg) throws IOException {
return -1;
}
if (!cfg.outputFormat.equalsIgnoreCase("hudi")) {
Review comment:
Sure. Will fix this once #1404 merged, with explicit format checking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] xushiyan commented on a change in pull request
#1405: [HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r393872943
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -45,41 +50,65 @@
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.execution.datasources.DataSource;
-
-import scala.Tuple2;
-import scala.collection.JavaConversions;
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
/**
* Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
*
* @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
*/
public class HoodieSnapshotExporter {
+
+ @FunctionalInterface
+ public interface Partitioner {
+
+ DataFrameWriter<Row> partition(Dataset<Row> source);
+
+ }
+
private static final Logger LOG = LogManager.getLogger(HoodieSnapshotExporter.class);
+ public static class OutputFormatValidator implements IValueValidator<String> {
+
+ static final String HUDI = "hudi";
+ static final List<String> FORMATS = ImmutableList.of("json", "parquet", HUDI);
Review comment:
@leesf Sure add some explanation in the CLI help line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `60.86%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
- Coverage 67.69% 67.69% -0.01%
- Complexity 243 244 +1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11084 +1
- Misses 4548 4550 +2
Partials 740 740
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `75.72% <60.86%> (-1.28%)` | `21 <3> (+1)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...c6cbfca](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392595777
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -47,36 +49,47 @@
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.execution.datasources.DataSource;
-import scala.Tuple2;
-import scala.collection.JavaConversions;
-
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
/**
* Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
*
* @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
*/
public class HoodieSnapshotExporter {
+
+ @FunctionalInterface
+ public interface Partitioner {
+
+ DataFrameWriter<Row> partition(Dataset<Row> source);
+
+ }
Review comment:
Also need UT for user-defined Partitioner?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1405:
[HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#issuecomment-599250128
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=h1) Report
> Merging [#1405](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/23afe7a4872fca66d9aeb36d209c6538a17d81f1&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `86.95%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1405 +/- ##
============================================
+ Coverage 67.69% 67.71% +0.01%
- Complexity 243 244 +1
============================================
Files 338 338
Lines 16371 16374 +3
Branches 1672 1674 +2
============================================
+ Hits 11083 11088 +5
+ Misses 4548 4546 -2
Partials 740 740
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `81.55% <86.95%> (+4.55%)` | `21.00 <3.00> (+1.00)` | |
| [...g/apache/hudi/metrics/InMemoryMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1405/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9Jbk1lbW9yeU1ldHJpY3NSZXBvcnRlci5qYXZh) | `25.00% <0.00%> (-50.00%)` | `0.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=footer). Last update [23afe7a...a5b8310](https://codecov.io/gh/apache/incubator-hudi/pull/1405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] xushiyan commented on a change in pull request
#1405: [HUDI-344] Add partitioner param to Exporter
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1405: [HUDI-344] Add partitioner param to Exporter
URL: https://github.com/apache/incubator-hudi/pull/1405#discussion_r392606711
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
##########
@@ -47,36 +49,47 @@
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.execution.datasources.DataSource;
-import scala.Tuple2;
-import scala.collection.JavaConversions;
-
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
+import scala.Tuple2;
+import scala.collection.JavaConversions;
+
/**
* Export the latest records of Hudi dataset to a set of external files (e.g., plain parquet files).
*
* @experimental This export is an experimental tool. If you want to export hudi to hudi, please use HoodieSnapshotCopier.
*/
public class HoodieSnapshotExporter {
+
+ @FunctionalInterface
+ public interface Partitioner {
+
+ DataFrameWriter<Row> partition(Dataset<Row> source);
+
+ }
Review comment:
Sure. Will fix this once #1404 merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services