[jira] [Commented] (HUDI-2023) Validate Schema evolution in hudi

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363761#comment-17363761 ] 

sivabalan narayanan commented on HUDI-2023:
-------------------------------------------

I tested both COW and MOR for simple schema evolution of adding a new column. Here are my findings. 

 

// "succeeds" refers to write succeeded and a read following the write succeeded to read entire dataset. 

 
|| ||COW||MOR||
|Adding a new nullable column at root level at the end|succeeds|succeeds|
|Adding a new nullable column to inner struct (at the end)|succeeds|succeeds|
|Adding a new non-nullable column at root level at the end|fails|write succeeds, but read fails as expected|
|Adding a new non-nullable column to inner struct (at the end)|fails|write succeeds, but read fails as expected|

 

Validated so far w/ spark datasource. Will update once I have results w/ delta streamer. 

 

 

> Validate Schema evolution in hudi
> ---------------------------------
>
>                 Key: HUDI-2023
>                 URL: https://issues.apache.org/jira/browse/HUDI-2023
>             Project: Apache Hudi
>          Issue Type: Test
>          Components: Testing
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Major
>
> Test schema evolution in hudi and document the same



--
This message was sent by Atlassian Jira
(v8.3.4#803005)