You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/04 21:02:29 UTC
[GitHub] [incubator-hudi] garyli1019 opened a new pull request #1592: WIP[Hudi 69] Spark Datasource for MOR table
garyli1019 opened a new pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] bhasudha commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-629781283
@garyli1019 taking a look at the PR. will get back soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] xushiyan commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-629812714
@garyli1019 sadly to see this weird NPE persists. It would be helpful to have debug mode in travis and ssh into the container and then investigate.
https://docs.travis-ci.com/user/running-build-in-debug-mode/#enabling-debug-mode
> For public repositories, we have to enable it on a repository basis.
To enable debug for your public repositories, please email us at support@travis-ci.com and let us know which repositories you want activated.
@bhasudha @vinothchandar This is difficult to troubleshoot as local test is passing..as the docs described, could we try to enable this debugging feature?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] garyli1019 commented on pull request #1592: [Hudi-822] decouple Hudi related logics from HoodieInputFormat
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-632977748
Folks, put these two commits together is too difficult to review. Let's do one step at a time.
1. This PR is decoupling Hudi related logics from HoodieInputFormat.
2. After this we can do Spark Datasource on MOR.
3. Then Spark Datasource incremental view on MOR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] garyli1019 commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-628828782
Hello @xushiyan , may I ask a question regarding the functional testing? I believe you are the expert on this topic in our community :)
This PR passed CI in my local but failed on Travis.
`java.lang.NullPointerException at org.apache.hudi.functional.TestDataSource.testSparkDatasourceForMergeOnRead(TestDataSource.scala:227)`. Do you know where should I look into this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] garyli1019 commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-629011466
Thanks @xushiyan! I will give it a try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] garyli1019 commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-624241028
Hi @vinothchandar @bhasudha this PR is ready for review. The CI failure was strange because it passed in my local. I will take a closer look here.
To review this bulky PR, I'd recommend to:
- review two commits separately.
- See the implementation session of RFC. I added the workflow of how the records were loaded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-commenter commented on pull request #1592: [Hudi-822] decouple Hudi related logics from HoodieInputFormat
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-632985999
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=h1) Report
> Merging [#1592](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/f34de3fb2738c8c36c937eba8df2a6848fafa886&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1592 +/- ##
============================================
- Coverage 18.21% 18.16% -0.06%
- Complexity 856 857 +1
============================================
Files 348 351 +3
Lines 15334 15388 +54
Branches 1523 1525 +2
============================================
+ Hits 2793 2795 +2
- Misses 12184 12236 +52
Partials 357 357
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...g/apache/hudi/hadoop/HoodieParquetInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVBhcnF1ZXRJbnB1dEZvcm1hdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVJPVGFibGVQYXRoRmlsdGVyLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hadoop/realtime/AbstractRealtimeRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0Fic3RyYWN0UmVhbHRpbWVSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...oop/realtime/HoodieParquetRealtimeInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVBhcnF1ZXRSZWFsdGltZUlucHV0Rm9ybWF0LmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...a/org/apache/hudi/hadoop/utils/HoodieHiveUtil.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUhpdmVVdGlsLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/hadoop/utils/HoodieInputFormatUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUlucHV0Rm9ybWF0VXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...i/hadoop/utils/HoodieRealtimeInputFormatUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lSW5wdXRGb3JtYXRVdGlscy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [5 more](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=footer). Last update [f34de3f...10109f9](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-commenter edited a comment on pull request #1592: [Hudi-822] decouple Hudi related logics from HoodieInputFormat
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-632985999
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=h1) Report
> Merging [#1592](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/f34de3fb2738c8c36c937eba8df2a6848fafa886&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1592 +/- ##
============================================
- Coverage 18.21% 18.16% -0.06%
- Complexity 856 857 +1
============================================
Files 348 351 +3
Lines 15334 15388 +54
Branches 1523 1525 +2
============================================
+ Hits 2793 2795 +2
- Misses 12184 12236 +52
Partials 357 357
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...g/apache/hudi/hadoop/HoodieParquetInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVBhcnF1ZXRJbnB1dEZvcm1hdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVJPVGFibGVQYXRoRmlsdGVyLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hadoop/realtime/AbstractRealtimeRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0Fic3RyYWN0UmVhbHRpbWVSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...oop/realtime/HoodieParquetRealtimeInputFormat.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVBhcnF1ZXRSZWFsdGltZUlucHV0Rm9ybWF0LmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
| [...a/org/apache/hudi/hadoop/utils/HoodieHiveUtil.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUhpdmVVdGlsLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/hadoop/utils/HoodieInputFormatUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUlucHV0Rm9ybWF0VXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...i/hadoop/utils/HoodieRealtimeInputFormatUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lSW5wdXRGb3JtYXRVdGlscy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [5 more](https://codecov.io/gh/apache/incubator-hudi/pull/1592/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=footer). Last update [f34de3f...10109f9](https://codecov.io/gh/apache/incubator-hudi/pull/1592?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] xushiyan commented on pull request #1592: [Hudi-69] Spark Datasource for MOR table
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #1592:
URL: https://github.com/apache/incubator-hudi/pull/1592#issuecomment-628845961
@garyli1019 it does look very weird... getting NPE at org.apache.hudi.functional.TestDataSource.testSparkDatasourceForMergeOnRead(TestDataSource.scala:227, means the `basepath` was set to null somehow. But wouldn't it be null in L222 and L223 of the same method? The test method passes locally for me as well. I don't have clue on why travis ran it differently..but at least try this change for that method
```java
@Test
def testSparkDatasourceForMergeOnRead(@TempDir tempDir: java.nio.file.Path) {
val basePath = tempDir.toAbsolutePath.toString
}
```
junit 5 should give it a new dedicated temp dir.. see if NPE still persists..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org