[GitHub] [hudi] aditiwari01 edited a comment on issue #2802: Hive read issues when different partition have different schemas.

[GitBox <gi...@apache.org>]

aditiwari01 edited a comment on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-819561416


   Hi @n3nash 
   
   The issue I am facing is in the case when commit gets succeeded. Let me explain the issue with an example:
   
   Commit1: insert key1 in partition1 and key2 in partition2 with schema1.
   Commit2: update key2 in partition2 with schema2
   
   In this case the schema2 is registered successfully after commit2. But the issue is that the `AbstractRealtimeRecordReader` is not reading the latest schema of the table. `AbstractRealtimeRecordReader` tries to read the latest schema for each partition from that last edited log/parquet file.
   
   As for the given example, the last edit in partition1 was with schema1. Hence the above code snippet that I have pasted in first comment, takes inn schema1 as writerSchema, causing the issue. (when split belongs to partition1)
   
   Also I replace the said logic with taking writerSchema from TableSchemaResolver rather than log/parquet file and it solves the issue for me.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org