[jira] [Commented] (HUDI-3467) Check shutdown logic with async compaction in Spark Structured Streaming

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-3467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499172#comment-17499172 ] 

sivabalan narayanan commented on HUDI-3467:
-------------------------------------------

Update: 

I confirmed that when regular writes fail, async table services shuts down. and when async table services shuts down, regular writes fail. 

But just that I could not exit from the spark-shell. It is just stuck. I had to use `kill -9 pid` to kill the process altogether. But chances that spark-streaming works that way. since reading from df stream is continuous. 

This is not an immediate necessity to fix, since both of them gets shutdown on any issue. 

 

> Check shutdown logic with async compaction in Spark Structured Streaming
> ------------------------------------------------------------------------
>
>                 Key: HUDI-3467
>                 URL: https://issues.apache.org/jira/browse/HUDI-3467
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: compaction, spark
>            Reporter: Ethan Guo
>            Assignee: sivabalan narayanan
>            Priority: Critical
>              Labels: hudi-on-call
>             Fix For: 0.11.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)