You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/04/20 15:54:55 UTC
[GitHub] [incubator-hudi] pratyakshsharma opened a new pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
pratyakshsharma opened a new pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#discussion_r412927375
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -60,7 +61,7 @@
private static ThreadLocal<BinaryDecoder> reuseDecoder = ThreadLocal.withInitial(() -> null);
// All metadata fields are optional strings.
- private static final Schema METADATA_FIELD_SCHEMA =
+ public static final Schema METADATA_FIELD_SCHEMA =
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-628611119
use of JsonProperties.NULL_VALUE makes sense rather than passing null. Let me do this change at few other places as well. Rest looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631444156
@pratyakshsharma by close, you mean final review and merge right? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#discussion_r424807539
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -60,7 +61,7 @@
private static ThreadLocal<BinaryDecoder> reuseDecoder = ThreadLocal.withInitial(() -> null);
// All metadata fields are optional strings.
- private static final Schema METADATA_FIELD_SCHEMA =
+ protected static final Schema METADATA_FIELD_SCHEMA =
Review comment:
just removing `private` should make this easy for testing... i.e package private as opposed to protected?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-io edited a comment on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616904151
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report
> Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/09fd6f64c527e6a822c4e17dc4e61b8fdee28189&el=desc) will **decrease** coverage by `55.62%`.
> The diff coverage is `8.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1538 +/- ##
=============================================
- Coverage 72.32% 16.69% -55.63%
- Complexity 294 796 +502
=============================================
Files 374 340 -34
Lines 16366 15025 -1341
Branches 1649 1499 -150
=============================================
- Hits 11836 2509 -9327
- Misses 3798 12186 +8388
+ Partials 732 330 -402
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../org/apache/hudi/client/CompactionAdminClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0NvbXBhY3Rpb25BZG1pbkNsaWVudC5qYXZh) | `0.00% <ø> (-70.62%)` | `0.00 <0.00> (ø)` | |
| [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00 <0.00> (ø)` | |
| [...org/apache/hudi/client/utils/SparkConfigUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L3V0aWxzL1NwYXJrQ29uZmlnVXRpbHMuamF2YQ==) | `8.00% <0.00%> (-77.72%)` | `2.00 <0.00> (+2.00)` | :arrow_down: |
| [...org/apache/hudi/config/HoodieCompactionConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZUNvbXBhY3Rpb25Db25maWcuamF2YQ==) | `54.90% <ø> (-26.48%)` | `3.00 <0.00> (+3.00)` | :arrow_down: |
| [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `43.29% <0.00%> (-41.36%)` | `47.00 <0.00> (+47.00)` | :arrow_down: |
| [...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00 <0.00> (ø)` | |
| [...c/main/java/org/apache/hudi/index/HoodieIndex.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvSG9vZGllSW5kZXguamF2YQ==) | `41.17% <ø> (-47.06%)` | `3.00 <0.00> (+3.00)` | :arrow_down: |
| [...pache/hudi/index/bloom/HoodieGlobalBloomIndex.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvYmxvb20vSG9vZGllR2xvYmFsQmxvb21JbmRleC5qYXZh) | `0.00% <0.00%> (-91.67%)` | `0.00 <0.00> (ø)` | |
| [...n/java/org/apache/hudi/index/hbase/HBaseIndex.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvaGJhc2UvSEJhc2VJbmRleC5qYXZh) | `0.00% <0.00%> (-83.26%)` | `0.00 <0.00> (ø)` | |
| [...n/java/org/apache/hudi/io/AppendHandleFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vQXBwZW5kSGFuZGxlRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [392 more](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=footer). Last update [09fd6f6...00f6ef8](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631472874
> @pratyakshsharma by close, you mean final review and merge right? :)
Yes :)
Rebased and pushed again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631445518
@pratyakshsharma could you rebase again and repush .. codecov seems to need that to work..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-617566396
@pratyakshsharma Got it.. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-io commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616904151
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report
> Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/09fd6f64c527e6a822c4e17dc4e61b8fdee28189&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1538 +/- ##
============================================
+ Coverage 72.32% 72.39% +0.07%
Complexity 294 294
============================================
Files 374 374
Lines 16366 16379 +13
Branches 1649 1651 +2
============================================
+ Hits 11836 11858 +22
+ Misses 3798 3791 -7
+ Partials 732 730 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `95.19% <100.00%> (+2.05%)` | `0.00 <0.00> (ø)` | |
| [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `84.84% <0.00%> (+0.19%)` | `0.00% <0.00%> (ø%)` | |
| [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `56.70% <0.00%> (+6.13%)` | `0.00% <0.00%> (ø%)` | |
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `89.65% <0.00%> (+10.34%)` | `0.00% <0.00%> (ø%)` | |
| [...e/hudi/exception/SchemaCompatabilityException.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL1NjaGVtYUNvbXBhdGFiaWxpdHlFeGNlcHRpb24uamF2YQ==) | `33.33% <0.00%> (+33.33%)` | `0.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=footer). Last update [09fd6f6...b13db4d](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-629419001
@vinothchandar We can close this now :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-617743201
> I need to wrap my head around some of this myself.. So please give me sometime to review..
Sure. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616649362
Few observations related to issues we have faced recently:
1. If we specify \"default\": null in string schema for a field or specify NullNode.getInstance() for default value when defining a field and then invoke field.defaultValue(), it returns a NullNode instance which in turn gives JsonProperties.Null instance when field.defaultVal() is invoked.
When we try to validate such a record in rewrite() function in HoodieAvroUtils class, the validate function internally tries to resolve a union type schema. At this point JsonProperties.Null type value is not handled. Rest of the data types are handled.
Rest all the cases, defaultVal() either returns proper data type or it comes as null i.e defaultValue variable of Field class is null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar merged pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616904151
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report
> Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/09fd6f64c527e6a822c4e17dc4e61b8fdee28189&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1538 +/- ##
============================================
+ Coverage 72.32% 72.39% +0.07%
Complexity 294 294
============================================
Files 374 374
Lines 16366 16379 +13
Branches 1649 1651 +2
============================================
+ Hits 11836 11858 +22
+ Misses 3798 3791 -7
+ Partials 732 730 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `95.19% <100.00%> (+2.05%)` | `0.00 <0.00> (ø)` | |
| [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZVdyaXRlQ29uZmlnLmphdmE=) | `84.84% <0.00%> (+0.19%)` | `0.00% <0.00%> (ø%)` | |
| [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `56.70% <0.00%> (+6.13%)` | `0.00% <0.00%> (ø%)` | |
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `89.65% <0.00%> (+10.34%)` | `0.00% <0.00%> (ø%)` | |
| [...e/hudi/exception/SchemaCompatabilityException.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL1NjaGVtYUNvbXBhdGFiaWxpdHlFeGNlcHRpb24uamF2YQ==) | `33.33% <0.00%> (+33.33%)` | `0.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=footer). Last update [09fd6f6...b13db4d](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631453314
> @pratyakshsharma could you rebase again and repush .. codecov seems to need that to work..
Ack.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#discussion_r412689006
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -214,7 +215,11 @@ private static GenericRecord rewrite(GenericRecord record, LinkedHashSet<Field>
GenericRecord newRecord = new GenericData.Record(newSchema);
for (Schema.Field f : fieldsToWrite) {
if (record.get(f.name()) == null) {
- newRecord.put(f.name(), f.defaultVal());
+ if (f.defaultVal() instanceof JsonProperties.Null) {
Review comment:
This code change is exactly what @afilipchik did in his PR :)
I have just added some more test cases to be sure nothing breaks around default values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-commenter commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631510907
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report
> Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/74ecc27e920c70fa4598d8e5a696954203a5b127&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `50.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1538 +/- ##
============================================
- Coverage 18.34% 18.33% -0.02%
- Complexity 854 855 +1
============================================
Files 344 344
Lines 15172 15167 -5
Branches 1512 1512
============================================
- Hits 2784 2781 -3
+ Misses 12035 12033 -2
Partials 353 353
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `48.09% <50.00%> (-1.91%)` | `22.00 <0.00> (ø)` | |
| [...apache/hudi/common/fs/HoodieWrapperFileSystem.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0hvb2RpZVdyYXBwZXJGaWxlU3lzdGVtLmphdmE=) | `22.69% <0.00%> (+0.70%)` | `29.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=footer). Last update [74ecc27...1373dee](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-628412207
Okay did that as well.. @pratyakshsharma let me know if you agree.. We can land after CI passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#discussion_r412647196
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -214,7 +215,11 @@ private static GenericRecord rewrite(GenericRecord record, LinkedHashSet<Field>
GenericRecord newRecord = new GenericData.Record(newSchema);
for (Schema.Field f : fieldsToWrite) {
if (record.get(f.name()) == null) {
- newRecord.put(f.name(), f.defaultVal());
+ if (f.defaultVal() instanceof JsonProperties.Null) {
Review comment:
@afilipchik to get your eyes on this as well ..
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -60,7 +61,7 @@
private static ThreadLocal<BinaryDecoder> reuseDecoder = ThreadLocal.withInitial(() -> null);
// All metadata fields are optional strings.
- private static final Schema METADATA_FIELD_SCHEMA =
+ public static final Schema METADATA_FIELD_SCHEMA =
Review comment:
this need not be public right.. just being package protected will allow you to access this in tests?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-616650031
@vinothchandar Please take a pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] codecov-commenter edited a comment on pull request #1538: [HUDI-803]: added more test cases in TestHoodieAvroUtils.class
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1538:
URL: https://github.com/apache/incubator-hudi/pull/1538#issuecomment-631510907
# [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=h1) Report
> Merging [#1538](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-hudi/commit/74ecc27e920c70fa4598d8e5a696954203a5b127&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `50.00%`.
[](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1538 +/- ##
============================================
- Coverage 18.34% 18.33% -0.02%
- Complexity 854 855 +1
============================================
Files 344 344
Lines 15172 15167 -5
Branches 1512 1512
============================================
- Hits 2784 2781 -3
+ Misses 12035 12033 -2
Partials 353 353
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ain/java/org/apache/hudi/avro/HoodieAvroUtils.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXZyby9Ib29kaWVBdnJvVXRpbHMuamF2YQ==) | `48.09% <50.00%> (-1.91%)` | `22.00 <0.00> (ø)` | |
| [...apache/hudi/common/fs/HoodieWrapperFileSystem.java](https://codecov.io/gh/apache/incubator-hudi/pull/1538/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0hvb2RpZVdyYXBwZXJGaWxlU3lzdGVtLmphdmE=) | `22.69% <0.00%> (+0.70%)` | `29.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=footer). Last update [74ecc27...1373dee](https://codecov.io/gh/apache/incubator-hudi/pull/1538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org