You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/04/02 18:23:33 UTC
[GitHub] [incubator-hudi] satishkotha opened a new pull request #1479:
[HUDI-758] Modify Integration test to include incremental queries on MOR
tables
satishkotha opened a new pull request #1479: [HUDI-758] Modify Integration test to include incremental queries on MOR tables
URL: https://github.com/apache/incubator-hudi/pull/1479
## What is the purpose of the pull request
Extend IntegrationTest to do incremental reads on MOR tables and validate responses
## Brief change log
Extend IntegrationTest to do incremental reads on MOR tables and validate responses
## Verify this pull request
ITTestHoodieDemo only change.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1479: [HUDI-758]
Modify Integration test to include incremental queries on MOR tables
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1479: [HUDI-758] Modify Integration test to include incremental queries on MOR tables
URL: https://github.com/apache/incubator-hudi/pull/1479#issuecomment-608922301
@bhasudha could you please take a look? this is relevant to wht you are fixing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bhasudha merged pull request #1479: [HUDI-758]
Modify Integration test to include incremental queries on MOR tables
Posted by GitBox <gi...@apache.org>.
bhasudha merged pull request #1479: [HUDI-758] Modify Integration test to include incremental queries on MOR tables
URL: https://github.com/apache/incubator-hudi/pull/1479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] codecov-io commented on issue #1479: [HUDI-758]
Modify Integration test to include incremental queries on MOR tables
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1479: [HUDI-758] Modify Integration test to include incremental queries on MOR tables
URL: https://github.com/apache/incubator-hudi/pull/1479#issuecomment-611277336
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=h1) Report
> Merging [#1479](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/bd716ece1826fcc44092bd0add6cbbc097eeaec2&el=desc) will **increase** coverage by `71.02%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1479 +/- ##
=============================================
+ Coverage 0.62% 71.65% +71.02%
- Complexity 2 290 +288
=============================================
Files 299 338 +39
Lines 14645 15931 +1286
Branches 1483 1625 +142
=============================================
+ Hits 92 11415 +11323
+ Misses 14550 3784 -10766
- Partials 3 732 +729
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...java/org/apache/hudi/client/HoodieCleanClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUNsZWFuQ2xpZW50LmphdmE=) | | | |
| [...va/org/apache/hudi/avro/MercifulJsonConverter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9NZXJjaWZ1bEpzb25Db252ZXJ0ZXIuamF2YQ==) | | | |
| [...java/com/uber/hoodie/hadoop/HoodieInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9jb20vdWJlci9ob29kaWUvaGFkb29wL0hvb2RpZUlucHV0Rm9ybWF0LmphdmE=) | | | |
| [...die/hadoop/realtime/HoodieRealtimeInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9jb20vdWJlci9ob29kaWUvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVJlYWx0aW1lSW5wdXRGb3JtYXQuamF2YQ==) | | | |
| [...ava/org/apache/hudi/metrics/JmxReporterServer.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9KbXhSZXBvcnRlclNlcnZlci5qYXZh) | | | |
| [...g/apache/hudi/metrics/MetricsGraphiteReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzR3JhcGhpdGVSZXBvcnRlci5qYXZh) | | | |
| [...c/main/java/org/apache/hudi/table/CleanHelper.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvQ2xlYW5IZWxwZXIuamF2YQ==) | | | |
| [...va/org/apache/hudi/metrics/JmxMetricsReporter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9KbXhNZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | | | |
| [...java/org/apache/hudi/common/HoodieJsonPayload.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL0hvb2RpZUpzb25QYXlsb2FkLmphdmE=) | | | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `100.00% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| ... and [307 more](https://codecov.io/gh/apache/incubator-hudi/pull/1479/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=footer). Last update [bd716ec...c0f6f42](https://codecov.io/gh/apache/incubator-hudi/pull/1479?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bhasudha commented on a change in pull request
#1479: [HUDI-758] Modify Integration test to include incremental queries on
MOR tables
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #1479: [HUDI-758] Modify Integration test to include incremental queries on MOR tables
URL: https://github.com/apache/incubator-hudi/pull/1479#discussion_r405870533
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
##########
@@ -267,23 +270,37 @@ private void testSparkSQLAfterSecondBatch() throws Exception {
assertStdOutContains(stdOutErrPair, "|GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|");
}
- private void testIncrementalHiveQuery() throws Exception {
+ private void testIncrementalHiveQuery(String minCommitTimeScript, String incrementalCommandsFile,
+ String expectedOutput, int expectedTimes) throws Exception {
String minCommitTime =
- executeCommandStringInDocker(ADHOC_2_CONTAINER, MIN_COMMIT_TIME_SCRIPT, true).getStdout().toString();
+ executeCommandStringInDocker(ADHOC_2_CONTAINER, minCommitTimeScript, true).getStdout().toString();
Pair<String, String> stdOutErrPair =
- executeHiveCommandFile(HIVE_INCREMENTAL_COMMANDS, "min.commit.time=" + minCommitTime + "`");
- assertStdOutContains(stdOutErrPair, "| GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |");
+ executeHiveCommandFile(incrementalCommandsFile, "min.commit.time=" + minCommitTime + "`");
+ assertStdOutContains(stdOutErrPair, expectedOutput, expectedTimes);
+ }
+
+ private void testIncrementalHiveQueryBeforeCompaction() throws Exception {
+ String expectedOutputCOW = "| GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |";
Review comment:
minor nit: please consider renaming the variable to `expectedOutput` instead since its used across all types of tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services