You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/31 03:02:17 UTC
[GitHub] [hudi] shenh062326 opened a new pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
shenh062326 opened a new pull request #1690:
URL: https://github.com/apache/hudi/pull/1690
## What is the purpose of the pull request
* For HUDI-908, first add decimals to HoodieTestDataGenerator.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r436311736
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
> Thanks for the explanation. Makes sense. ObjectMapper uses List to represent FixedType. So, each entry in the list is within the range of [-128, 127]. right ?
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435744173
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
@bvaradar Is it a bug?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some bugs
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r443216067
##########
File path: hudi-utilities/src/test/resources/delta-streamer-config/source.avsc
##########
@@ -43,8 +43,41 @@
}, {
"name" : "end_lon",
"type" : "double"
- },
- {
+ }, {
+ "name" : "int_val",
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some bugs
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-646971258
@bvaradar I have add all the missing types and fix some bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r436313087
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637973048
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=h1) Report
> Merging [#1690](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a9a97d6af47841caaa745497ec425267db0873c8&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1690 +/- ##
============================================
+ Coverage 18.18% 18.20% +0.01%
- Complexity 856 858 +2
============================================
Files 348 348
Lines 15351 15358 +7
Branches 1524 1525 +1
============================================
+ Hits 2792 2796 +4
- Misses 12202 12204 +2
- Partials 357 358 +1
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2VtYmVkZGVkL0VtYmVkZGVkVGltZWxpbmVTZXJ2aWNlLmphdmE=) | `72.22% <0.00%> (-2.07%)` | `7.00% <0.00%> (ø%)` | |
| [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `62.68% <0.00%> (-0.81%)` | `9.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...on/rollback/CopyOnWriteRollbackActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL3JvbGxiYWNrL0NvcHlPbldyaXRlUm9sbGJhY2tBY3Rpb25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...on/rollback/MergeOnReadRollbackActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL3JvbGxiYWNrL01lcmdlT25SZWFkUm9sbGJhY2tBY3Rpb25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...apache/hudi/common/fs/HoodieWrapperFileSystem.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0hvb2RpZVdyYXBwZXJGaWxlU3lzdGVtLmphdmE=) | `22.69% <0.00%> (+0.70%)` | `29.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=footer). Last update [a9a97d6...698269d](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-641301829
@shenh062326 : It makes sense to cover other data-types in a single PR. Can you also add them to this PR. Also, Can you let us know what the missing data types are ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435692529
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
1. Currently Fix.toString() will convert bytes to string,
```
public String toString() { return Arrays.toString(bytes); }
```
the result is like "decimal_val": [0, 0, 14, -63, -52],
2. MercifulJsonConverter.convert will parse json string to jsonObjectMap,
```
public GenericRecord convert(String json, Schema schema) {
try {
Map<String, Object> jsonObjectMap = mapper.readValue(json, Map.class);
return convertJsonToAvro(jsonObjectMap, schema);
} catch (IOException e) {
throw new HoodieIOException(e.getMessage(), e);
}
}
```
it will convert "decimal_val": [0, 0, 14, -63, -52], to ArrayList<Integer>.
3. the generateFixedTypeHandler() will process it incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some bugs
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r443164942
##########
File path: hudi-utilities/src/test/resources/delta-streamer-config/source.avsc
##########
@@ -43,8 +43,41 @@
}, {
"name" : "end_lon",
"type" : "double"
- },
- {
+ }, {
+ "name" : "int_val",
Review comment:
Minor: Instead of naming based on types, can we give relevant names like :distance_in_meters (int_type), seconds_since_epoch(long_val), payment_in_dollars,...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar merged pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some bugs
Posted by GitBox <gi...@apache.org>.
bvaradar merged pull request #1690:
URL: https://github.com/apache/hudi/pull/1690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435692529
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Currently Fix.toString() will convert bytes to string,
```
public String toString() { return Arrays.toString(bytes); }
```
the result is like "decimal_val": [0, 0, 14, -63, -52], the generateFixedTypeHandler() will process it incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator and fix some bugs
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-645150017
@bvaradar Can you take a look at this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637973048
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=h1) Report
> Merging [#1690](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a9a97d6af47841caaa745497ec425267db0873c8&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1690 +/- ##
=========================================
Coverage 18.18% 18.18%
Complexity 856 856
=========================================
Files 348 348
Lines 15351 15351
Branches 1524 1524
=========================================
Hits 2792 2792
Misses 12202 12202
Partials 357 357
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=footer). Last update [a9a97d6...4969d23](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 edited a comment on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 edited a comment on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-641671278
> @shenh062326 : It makes sense to cover other data-types in a single PR. Can you also add them to this PR. Also, Can you let us know what the missing data types are ?
The missing spark types:
```
BinaryType
IntegerType
LongType
FloatType
ByteType
ShortType
DecimalType
TimestampType
DateType
```
The missing the avro types:
```
FIXED,
BYTES,
INT,
LONG,
FLOAT
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r436257453
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Thanks for the explanation. Makes sense. ObjectMapper uses List<Integer> to represent FixedType. So, each entry in the list is within the range of [-128, 127]. right ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Minor Comment: Can you use List<Integer> instead of ArrayList<Integer>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435744173
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
@bvaradar Is it a bug? or maybe we shoud support both "decimal_val": [0, 0, 14, -63, -52] and "decimal_val": 0.123456
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-643685024
> > @shenh062326 : It makes sense to cover other data-types in a single PR. Can you also add them to this PR. Also, Can you let us know what the missing data types are ?
>
> The missing spark types:
>
> ```
> BinaryType
> IntegerType
> LongType
> FloatType
> ByteType
> ShortType
> DecimalType
> TimestampType
> DateType
> ```
>
> The missing the avro types:
>
> ```
> FIXED,
> BYTES,
> INT,
> LONG,
> FLOAT
> ```
Thanks. It makes sense to cover these types in the same PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637961149
> @shenh062326 I think you might want to resolve the conflicts.
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435744173
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
@bvaradar Is it a bug?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
bhasudha commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637287662
@shenh062326 I think you might want to resolve the conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-641671278
> @shenh062326 : It makes sense to cover other data-types in a single PR. Can you also add them to this PR. Also, Can you let us know what the missing data types are ?
The missing spark types:
```
BinaryType
IntegerType
LongType
FloatType
ByteType
ShortType
DecimalType
TimestampType
DateType
```
The missing the avro types:
```
ENUM,
FIXED,
BYTES,
INT,
LONG,
FLOAT,
NULL
```
But I am not sure if enum and null types are needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r436313137
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
> Minor Comment: Can you use List instead of ArrayList
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637973048
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=h1) Report
> Merging [#1690](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a9a97d6af47841caaa745497ec425267db0873c8&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1690 +/- ##
============================================
+ Coverage 18.18% 18.20% +0.01%
- Complexity 856 858 +2
============================================
Files 348 348
Lines 15351 15358 +7
Branches 1524 1525 +1
============================================
+ Hits 2792 2796 +4
- Misses 12202 12204 +2
- Partials 357 358 +1
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2VtYmVkZGVkL0VtYmVkZGVkVGltZWxpbmVTZXJ2aWNlLmphdmE=) | `72.22% <0.00%> (-2.07%)` | `7.00% <0.00%> (ø%)` | |
| [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `62.68% <0.00%> (-0.81%)` | `9.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...on/rollback/CopyOnWriteRollbackActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL3JvbGxiYWNrL0NvcHlPbldyaXRlUm9sbGJhY2tBY3Rpb25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...on/rollback/MergeOnReadRollbackActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL3JvbGxiYWNrL01lcmdlT25SZWFkUm9sbGJhY2tBY3Rpb25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...apache/hudi/common/fs/HoodieWrapperFileSystem.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0hvb2RpZVdyYXBwZXJGaWxlU3lzdGVtLmphdmE=) | `22.69% <0.00%> (+0.70%)` | `29.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=footer). Last update [a9a97d6...698269d](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435079221
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Is this guaranteed to be list for a fixed decimal value ? Shouldn't this be scalar ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1690: [HUDI-908] Add some data types to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637973048
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=h1) Report
> Merging [#1690](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a9a97d6af47841caaa745497ec425267db0873c8&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1690 +/- ##
============================================
- Coverage 18.18% 18.14% -0.04%
- Complexity 856 859 +3
============================================
Files 348 352 +4
Lines 15351 15411 +60
Branches 1524 1525 +1
============================================
+ Hits 2792 2797 +5
- Misses 12202 12256 +54
- Partials 357 358 +1
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2VtYmVkZGVkL0VtYmVkZGVkVGltZWxpbmVTZXJ2aWNlLmphdmE=) | `72.22% <0.00%> (-2.07%)` | `7.00% <0.00%> (ø%)` | |
| [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `36.06% <0.00%> (-1.87%)` | `9.00% <0.00%> (ø%)` | |
| [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `62.68% <0.00%> (-0.81%)` | `9.00% <0.00%> (+1.00%)` | :arrow_down: |
| [...org/apache/hudi/config/HoodieCompactionConfig.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlnL0hvb2RpZUNvbXBhY3Rpb25Db25maWcuamF2YQ==) | `55.33% <0.00%> (-0.67%)` | `3.00% <0.00%> (ø%)` | |
| [...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVJPVGFibGVQYXRoRmlsdGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...g/apache/hudi/hadoop/HoodieParquetInputFormat.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVBhcnF1ZXRJbnB1dEZvcm1hdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...he/hudi/table/action/commit/UpsertPartitioner.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9VcHNlcnRQYXJ0aXRpb25lci5qYXZh) | `55.39% <0.00%> (ø)` | `15.00% <0.00%> (ø%)` | |
| [...hudi/hadoop/hive/HoodieCombineHiveInputFormat.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2hpdmUvSG9vZGllQ29tYmluZUhpdmVJbnB1dEZvcm1hdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [.../hadoop/realtime/AbstractRealtimeRecordReader.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0Fic3RyYWN0UmVhbHRpbWVSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| ... and [13 more](https://codecov.io/gh/apache/hudi/pull/1690/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=footer). Last update [a9a97d6...e1e444a](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435692529
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
1. Currently Fix.toString() will convert bytes to string,
```
public String toString() { return Arrays.toString(bytes); }
```
the result is like "decimal_val": [0, 0, 14, -63, -52],
2. MercifulJsonConverter.convert will parse json string to jsonObjectMap,
```
public GenericRecord convert(String json, Schema schema) {
try {
Map<String, Object> jsonObjectMap = mapper.readValue(json, Map.class);
return convertJsonToAvro(jsonObjectMap, schema);
} catch (IOException e) {
throw new HoodieIOException(e.getMessage(), e);
}
}
```
it will convert "decimal_val": [0, 0, 14, -63, -52], to ArrayList<Integer>.
3. the generateFixedTypeHandler() will process it incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#discussion_r435692529
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java
##########
@@ -245,10 +245,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
return new JsonToAvroFieldProcessor() {
@Override
public Pair<Boolean, Object> convert(Object value, String name, Schema schema) {
- byte[] src = value.toString().getBytes();
+ ArrayList<Integer> converval = (ArrayList<Integer>) value;
Review comment:
Current Fix.toString() will convert bytes to string,
```
public String toString() { return Arrays.toString(bytes); }
```
the result is like "decimal_val": [0, 0, 14, -63, -52], the generateFixedTypeHandler() will process it incorrect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-640978958
@bvaradar Should I add all data types to this pr or open another pr. My original idea was that this pr fixes the bug of parsing decimal type, and another pr is added to add other data types
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #1690: [HUDI-908] Add decimals to HoodieTestDataGenerator
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1690:
URL: https://github.com/apache/hudi/pull/1690#issuecomment-637973048
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=h1) Report
> Merging [#1690](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/a9a97d6af47841caaa745497ec425267db0873c8&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1690 +/- ##
=========================================
Coverage 18.18% 18.18%
Complexity 856 856
=========================================
Files 348 348
Lines 15351 15351
Branches 1524 1524
=========================================
Hits 2792 2792
Misses 12202 12202
Partials 357 357
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=footer). Last update [a9a97d6...4969d23](https://codecov.io/gh/apache/hudi/pull/1690?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org