[jira] [Commented] (HUDI-2567) Verify synchronous metadata patch w/ multi writers end to end

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447520#comment-17447520 ] 

sivabalan narayanan commented on HUDI-2567:
-------------------------------------------

Scenarios tested:

 

1. Two concurrent writers writing to two different partitions. green. validated metadata file listing. 

2. two concurrent writers writing to two different file groups in same partition. green. validated metadata file listing. 

3. 3 concurrent writers. 2 of them overlap wrt file groups, while the other does not have any over lap. 2 out of 3 succeeded and 1 failed. green. validated metadata file listing. 

4. Deltastreamer and spark data source writer. spark-data source writer failed when it had overlapping writes wrt delta streamer commits. green. validated metadata file listing. 

5. Deltastreamer in continuous mode and async compaction. concurrent write from spark-datasource succeeds if started just after compaction got scheduled for same file group. green. 

6. Deltastreamer in continuous mode with async compaction and async clustering. Used allow updates strategy for clustering updates as I was focused on validating metadata file listing. Ran it for 20+ commits. all good. 

7. same set up as above, but crashed compaction 2 times. ran into the metadata inconsistent issue. but after bypassing that, things were good. regular delta commits were succeeding. and concurrent writer from spark-datasource succeeds if there is not over-lapping delta commits. if not, it fails. 

 

> Verify synchronous metadata patch w/ multi writers end to end
> -------------------------------------------------------------
>
>                 Key: HUDI-2567
>                 URL: https://issues.apache.org/jira/browse/HUDI-2567
>             Project: Apache Hudi
>          Issue Type: Sub-task
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Blocker
>              Labels: oct18_2021
>             Fix For: 0.10.0
>
>
> Verify synchronous metadata patch w/ multi writers end to end



--
This message was sent by Atlassian Jira
(v8.20.1#820001)