[GitHub] [hudi] codope commented on issue #3735: [SUPPORT] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column

[GitBox <gi...@apache.org>]

codope commented on issue #3735:
URL: https://github.com/apache/hudi/issues/3735#issuecomment-931133022


   I think this is a bug. What's happening here is that the [overwrite check](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java#L102) in OverwriteWithLatestAvroPayload is simply checking whether the two objects are equal or not. Since the nullable column gets converted to avro schema, the [defaultVal() method](https://avro.apache.org/docs/1.8.2/api/java/org/apache/avro/Schema.Field.html#defaultVal()) returns the corresponding `JsonProperties.Null` but the other object in the check is not the same type. So, the check returns false and hence that field gets overwritten. Instead we should modify that method to something like:
   ```
   public Boolean overwriteField(Object value, Object defaultValue) {
       if (value == null) {
         return defaultValue instanceof JsonProperties.Null;
       }
       return Objects.equals(value, defaultValue);
   }
   ```
   
   @peanut-chenzhong  I have filed [a bug](https://issues.apache.org/jira/browse/HUDI-2509). Please raise a PR if you have the fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org