[jira] [Resolved] (HUDI-1230) Spark-submit for MOR table creation via DataSource API hangs

["Vinoth Chandar (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinoth Chandar resolved HUDI-1230.
----------------------------------
    Resolution: Fixed

> Spark-submit for MOR table creation via DataSource API hangs
> ------------------------------------------------------------
>
>                 Key: HUDI-1230
>                 URL: https://issues.apache.org/jira/browse/HUDI-1230
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Compaction, Spark Integration
>            Reporter: Udit Mehrotra
>            Assignee: Udit Mehrotra
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.7.0
>
>
> After the addition of Async compaction functionality for streaming workloads with Hudi 0.6.0, MOR table creation via Hudi datasource hangs at the end of the job.
> This appears to be because of a bug where at [https://github.com/apache/hudi/blob/release-0.6.0/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L408] we are not checking if the async trigger function has been passed, which is passed only in case of streaming. As a result this becomes true for datasource as well.
> This leads to [https://github.com/apache/hudi/blob/release-0.6.0/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L421] not being triggered which does not stop the timeline server at the end, and as a result the job hangs and never exits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)