You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2023/06/17 21:09:32 UTC

[GitHub] [hudi] parisni commented on issue #9005: [SUPPORT]New merger fails with NPE when schema evolution with string occurs

parisni commented on issue #9005:
URL: https://github.com/apache/hudi/issues/9005#issuecomment-1595861748

   After some investigations, the legacy merger also fails in this particular case (see below log). For both, the reason is the new column added has `nullable=false`. Moreover, when turning `hoodie.avro.schema.validate=true` then it will fail early with error: `Incoming batch schema is not compatible with the table's one`.
   
   Then the NPE is likely not a real issue (but hard to understand for end user): we try to write a null in the new column for previous records in a not null column.
   
   Indeed, when the new column has `nullable=true`, then all goes fine with schema evolution. For example by inserting `df =spark.sql("select '2' as event_id, '2' as ts, '3' as version, 'foo' as event_date, case when 1=1 then 'foo' else null end as add_col")`
   
   What about providing the hability to turn the incomming schema in `nullable=true` automatically in hudi to avoid such errors ? @nsivabalan @yihua 
   
   ```
   Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old record into new file for key event_id:1 from old file /tmp/test_hudi_merger/version=3/event_date=foo/17d0126f-b7a8-4b7f-95e7-1e65a8f36e8d-0_0-83-83_20230617230002764.parquet to new file /tmp/test_hudi_merger/version=3/event_date=foo/17d0126f-b7a8-4b7f-95e7-1e65a8f36e8d-0_0-91-91_20230617230003209.parquet with writerSchema {
     "type" : "record",
     "name" : "test_hudi_merger_record",
     "namespace" : "hoodie.test_hudi_merger",
     "fields" : [ {
       "name" : "_hoodie_commit_time",
       "type" : [ "null", "string" ],
       "doc" : "",
       "default" : null
     }, {
       "name" : "_hoodie_commit_seqno",
       "type" : [ "null", "string" ],
       "doc" : "",
       "default" : null
     }, {
       "name" : "_hoodie_record_key",
       "type" : [ "null", "string" ],
       "doc" : "",
       "default" : null
     }, {
       "name" : "_hoodie_partition_path",
       "type" : [ "null", "string" ],
       "doc" : "",
       "default" : null
     }, {
       "name" : "_hoodie_file_name",
       "type" : [ "null", "string" ],
       "doc" : "",
       "default" : null
     }, {
       "name" : "event_id",
       "type" : "string"
     }, {
       "name" : "ts",
       "type" : "string"
     }, {
       "name" : "version",
       "type" : "string"
     }, {
       "name" : "event_date",
       "type" : "string"
     }, {
       "name" : "add_col",
       "type" : "string"
     } ]
   }
           at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:370)
           at org.apache.hudi.table.action.commit.BaseMergeHelper$UpdateHandler.consume(BaseMergeHelper.java:54)
           at org.apache.hudi.table.action.commit.BaseMergeHelper$UpdateHandler.consume(BaseMergeHelper.java:44)
           at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:67)
           ... 33 more
   Caused by: java.lang.RuntimeException: Null-value for required field: add_col
           at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:200)
           at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:171)
           at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
           at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:310)
           at org.apache.hudi.io.storage.HoodieBaseParquetWriter.write(HoodieBaseParquetWriter.java:80)
           at org.apache.hudi.io.storage.HoodieAvroParquetWriter.writeAvro(HoodieAvroParquetWriter.java:76)
           at org.apache.hudi.io.storage.HoodieAvroFileWriter.write(HoodieAvroFileWriter.java:51)
           at org.apache.hudi.io.storage.HoodieFileWriter.write(HoodieFileWriter.java:43)
           at org.apache.hudi.io.HoodieMergeHandle.writeToFile(HoodieMergeHandle.java:384)
           at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:365)
           ... 36 more
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org