[jira] [Commented] (HUDI-3799) Understand reason behind "Not an avro data file" with hudi

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519241#comment-17519241 ] 

sivabalan narayanan commented on HUDI-3799:
-------------------------------------------

We might need to fix how archival handles the empty completed instant. 
 * Re-create an valid completed instant (could be empty contents too), but lets not delete the entry. create a empty instant and archive it. so that we have a record that rollback. completed was part of the timeline.
 * Do this for any completed instant and not just for clean and rollback. bcoz, in hdfs and local, any completed instant could be empty during crash. 

> Understand reason behind "Not an avro data file" with hudi
> ----------------------------------------------------------
>
>                 Key: HUDI-3799
>                 URL: https://issues.apache.org/jira/browse/HUDI-3799
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> We merged [https://github.com/apache/hudi/pull/4016] to tackle "Not an avro data file" exception while cleaning or archiving. We need to understand why and when such exception happens. and try to mitigate it before happening if feasible. 
>  
> Atleast we should have a good understanding of what are the conditions under which this is expected. 
>  
> Ref: https://github.com/apache/hudi/pull/4016#pullrequestreview-841692564



--
This message was sent by Atlassian Jira
(v8.20.1#820001)