[jira] [Created] (HUDI-774) Spark to Avro converter incorrectly generates optional fields

["Alexander Filipchik (Jira)" <ji...@apache.org>]

Alexander Filipchik created HUDI-774:
----------------------------------------

             Summary: Spark to Avro converter incorrectly generates optional fields
                 Key: HUDI-774
                 URL: https://issues.apache.org/jira/browse/HUDI-774
             Project: Apache Hudi (incubating)
          Issue Type: Bug
            Reporter: Alexander Filipchik
             Fix For: 0.5.0


I think https://issues.apache.org/jira/browse/SPARK-28008 is a good descriptions of what is happening.

 

It can cause a situation when schema in the MOR log files is incompatible with the schema produced by RowBasedSchemaProvider, so compactions will stall.

 

I have a fix which is a bit hacky -> postprocess schema produced by the converter and

1) Make sure unions with null types have those null types at position 0

2) They have default values set to null

I couldn't find a way to do a clean fix as some classes that are problematic are from Hive and called from Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)