You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Danny Chen (Jira)" <ji...@apache.org> on 2021/11/27 05:19:00 UTC

[jira] [Updated] (HUDI-1765) disabling auto commit does not work with spark ds

     [ https://issues.apache.org/jira/browse/HUDI-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Chen updated HUDI-1765:
-----------------------------
    Fix Version/s:     (was: 0.10.0)

> disabling auto commit does not work with spark ds
> -------------------------------------------------
>
>                 Key: HUDI-1765
>                 URL: https://issues.apache.org/jira/browse/HUDI-1765
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Spark Integration
>            Reporter: sivabalan narayanan
>            Priority: Major
>              Labels: sev:high
>             Fix For: 0.11.0
>
>
> commitAndPerformPostOperations in HoodieSparkSqlWriter commits all operations w/o checking for autoCommitProp. So, this prop is not being honored at spark ds layer. 
>  
> Added below test to TestCOWDataSource and it fails. 
> [https://gist.github.com/nsivabalan/479320351bec95ba3e0c0dfa5abb2b06]
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)