You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@hudi.apache.org by "vbalaji@apache.org" <vb...@apache.org> on 2020/08/27 05:45:38 UTC

0.6.0 Bug : [HUDI-1230] Spark Data Source Batch Write on MOR table not shutting down

Dear Hudi Users,
We noticed an issue with 0.6.0 release and would like to notify you all.
https://issues.apache.org/jira/browse/HUDI-1230
This affects Spark Datasource batch writes on MOR tables only. This problem will NOT be seen when inline compaction is enabled. Spark Structured Streaming Writes on MOR table is also unaffected. 
The implication of this bug is that spark-submit jobs  running DataSource batch writes on MOR table will not shut down after the job is complete. 

As a work around, please set this redundant hoodie config in your spark-submit job when running batch writes in Spark.
hoodie.datasource.compaction.async.enable=false

As mentioned above, if inline compaction is enabled, the above workaround is not needed at all.
Udit has already opened a PR to address this issue : https://github.com/apache/hudi/pull/2046
Thanks,Balaji.V