You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Vinay (Jira)" <ji...@apache.org> on 2021/06/08 12:50:00 UTC
[jira] [Commented] (HUDI-1892) NullPointerException when using
OverwriteNonDefaultsWithLatestAvroPayload at hudi 0.9.0
[ https://issues.apache.org/jira/browse/HUDI-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359328#comment-17359328 ]
Vinay commented on HUDI-1892:
-----------------------------
Looking at the code this issue could be because the defaultValue is not empty but the actual value of the field is null (This could be the case if field schema is Union type like "null,string")
Creating a PR to use String.valueOf(value) in the below function to handle this edge case
{code:java}
public Boolean overwriteField(Object value, Object defaultValue) {
return defaultValue == null ? value == null : defaultValue.toString().equals(value.toString());
}
{code}
> NullPointerException when using OverwriteNonDefaultsWithLatestAvroPayload at hudi 0.9.0
> ---------------------------------------------------------------------------------------
>
> Key: HUDI-1892
> URL: https://issues.apache.org/jira/browse/HUDI-1892
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: shenbing
> Assignee: Vinay
> Priority: Major
>
> using compiled hudi 0.9.0 with hadoop3.0.0 and hive3.1.1 after resolving dependency conflicts,
> I import hudi-spark-bundle_2.11-0.9.0-SNAPSHOT.jar into my project.
> When I using OverwriteNonDefaultsWithLatestAvroPayload to update field with new value, I got the error.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to combine/merge new record with old value in storage, for new record {HoodieRecord{key=HoodieKey { recordKey=1 partitionPath=date=1}, currentLocation='HoodieRecordLocation {instantTime=20210510160355, fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}', newLocation='HoodieRecordLocation {instantTime=20210510160400, fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}'}}, old value {{"_hoodie_commit_time": "20210510160355", "_hoodie_commit_seqno": "20210510160355_0_50", "_hoodie_record_key": "1", "_hoodie_partition_path": "date=1", "_hoodie_file_name": "9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0_0-1502-1519_20210510160355.parquet", "uuid": "1", "name": "jerry", "age": 10, "date": "1", "update_time": "1"}}
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
> ... 34 more
> Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to combine/merge new record with old value in storage, for new record {HoodieRecord{key=HoodieKey { recordKey=1 partitionPath=date=1}, currentLocation='HoodieRecordLocation {instantTime=20210510160355, fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}', newLocation='HoodieRecordLocation {instantTime=20210510160400, fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}'}}, old value {{"_hoodie_commit_time": "20210510160355", "_hoodie_commit_seqno": "20210510160355_0_50", "_hoodie_record_key": "1", "_hoodie_partition_path": "date=1", "_hoodie_file_name": "9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0_0-1502-1519_20210510160355.parquet", "uuid": "1", "name": "jerry", "age": 10, "date": "1", "update_time": "1"}}
> at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:290)
> at org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
> at org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
> at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.apache.hudi.common.model.OverwriteWithLatestAvroPayload.overwriteField(OverwriteWithLatestAvroPayload.java:97)
> at org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload.lambda$combineAndGetUpdateValue$0(OverwriteNonDefaultsWithLatestAvroPayload.java:67)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload.combineAndGetUpdateValue(OverwriteNonDefaultsWithLatestAvroPayload.java:64)
> at org.apache.hudi.common.model.HoodieRecordPayload.combineAndGetUpdateValue(HoodieRecordPayload.java:81)
> at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:276)
> ... 8 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)