You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/06/07 04:44:18 UTC
[GitHub] [hudi] lw309637554 opened a new pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
lw309637554 opened a new pull request #1711:
URL: https://github.com/apache/hudi/pull/1711
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
Fields out of order in MOR mode when using Hive,with setting hoodie.realtime.merge.skip = true
## Brief change log
modify getWriterSchema in RealtimeUnmergedRecordReader to getHiveSchema
## Verify this pull request
This pull request is already covered by existing tests TestHoodieRealtimeRecordReader.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1711:
URL: https://github.com/apache/hudi/pull/1711
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1711:
URL: https://github.com/apache/hudi/pull/1711#issuecomment-640326551
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=h1) Report
> Merging [#1711](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/acb1ada2f756b49d9f9a0aa152f99fcc9e86dde7&el=desc) will **decrease** coverage by `54.06%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1711 +/- ##
=============================================
- Coverage 72.25% 18.19% -54.07%
- Complexity 294 857 +563
=============================================
Files 374 348 -26
Lines 16371 15358 -1013
Branches 1654 1525 -129
=============================================
- Hits 11829 2794 -9035
- Misses 3806 12206 +8400
+ Partials 736 358 -378
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (-96.97%)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/common/model/ActionType.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0FjdGlvblR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/hudi/exception/HoodieIOException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUlPRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=footer). Last update [acb1ada...3abe325](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1711:
URL: https://github.com/apache/hudi/pull/1711#issuecomment-640326551
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=h1) Report
> Merging [#1711](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/acb1ada2f756b49d9f9a0aa152f99fcc9e86dde7&el=desc) will **decrease** coverage by `54.05%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1711 +/- ##
=============================================
- Coverage 72.25% 18.20% -54.06%
- Complexity 294 858 +564
=============================================
Files 374 348 -26
Lines 16371 15361 -1010
Branches 1654 1525 -129
=============================================
- Hits 11829 2796 -9033
- Misses 3806 12207 +8401
+ Partials 736 358 -378
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (-96.97%)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/common/model/ActionType.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0FjdGlvblR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/hudi/exception/HoodieIOException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUlPRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=footer). Last update [acb1ada...132c811](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1711:
URL: https://github.com/apache/hudi/pull/1711#issuecomment-640326551
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=h1) Report
> Merging [#1711](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/acb1ada2f756b49d9f9a0aa152f99fcc9e86dde7&el=desc) will **decrease** coverage by `54.06%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1711 +/- ##
=============================================
- Coverage 72.25% 18.19% -54.07%
- Complexity 294 857 +563
=============================================
Files 374 348 -26
Lines 16371 15358 -1013
Branches 1654 1525 -129
=============================================
- Hits 11829 2794 -9035
- Misses 3806 12206 +8400
+ Partials 736 358 -378
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (-96.97%)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9NZXRyaWNzUmVwb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/common/model/ActionType.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0FjdGlvblR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...java/org/apache/hudi/io/HoodieRangeInfoHandle.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUmFuZ2VJbmZvSGFuZGxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../java/org/apache/hudi/hadoop/InputPathHandler.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0lucHV0UGF0aEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/hudi/exception/HoodieIOException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUlPRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...org/apache/hudi/table/action/commit/SmallFile.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9TbWFsbEZpbGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vS2V5UmFuZ2VMb29rdXBUcmVlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...g/apache/hudi/exception/HoodieInsertException.java](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZUluc2VydEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [336 more](https://codecov.io/gh/apache/hudi/pull/1711/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=footer). Last update [acb1ada...3abe325](https://codecov.io/gh/apache/hudi/pull/1711?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1711:
URL: https://github.com/apache/hudi/pull/1711#discussion_r436636431
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
##########
@@ -82,7 +82,7 @@ public RealtimeUnmergedRecordReader(HoodieRealtimeFileSplit split, JobConf job,
false, jobConf.getInt(MAX_DFS_STREAM_BUFFER_SIZE_PROP, DEFAULT_MAX_DFS_STREAM_BUFFER_SIZE), record -> {
// convert Hoodie log record to Hadoop AvroWritable and buffer
GenericRecord rec = (GenericRecord) record.getData().getInsertValue(getReaderSchema()).get();
- ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getWriterSchema());
+ ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getHiveSchema());
Review comment:
@vinothchandar This is a bug when using RealtimeUnmergedRecordReader, merged record reader works fine. unmerged record reader should use hive schema rather than writer schema here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1711: [HUDI-974] fix fields out of order in MOR mode when using Hive
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1711:
URL: https://github.com/apache/hudi/pull/1711#discussion_r436633121
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
##########
@@ -82,7 +82,7 @@ public RealtimeUnmergedRecordReader(HoodieRealtimeFileSplit split, JobConf job,
false, jobConf.getInt(MAX_DFS_STREAM_BUFFER_SIZE_PROP, DEFAULT_MAX_DFS_STREAM_BUFFER_SIZE), record -> {
// convert Hoodie log record to Hadoop AvroWritable and buffer
GenericRecord rec = (GenericRecord) record.getData().getInsertValue(getReaderSchema()).get();
- ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getWriterSchema());
+ ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getHiveSchema());
Review comment:
what do we use in the merged record reader? cc @bvaradar
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
##########
@@ -82,7 +82,7 @@ public RealtimeUnmergedRecordReader(HoodieRealtimeFileSplit split, JobConf job,
false, jobConf.getInt(MAX_DFS_STREAM_BUFFER_SIZE_PROP, DEFAULT_MAX_DFS_STREAM_BUFFER_SIZE), record -> {
// convert Hoodie log record to Hadoop AvroWritable and buffer
GenericRecord rec = (GenericRecord) record.getData().getInsertValue(getReaderSchema()).get();
- ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getWriterSchema());
+ ArrayWritable aWritable = (ArrayWritable) avroToArrayWritable(rec, getHiveSchema());
Review comment:
sg.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org