You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/09/25 06:08:00 UTC

[jira] [Updated] (SPARK-44462) Fix the session passed to foreachBatch.

     [ https://issues.apache.org/jira/browse/SPARK-44462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim updated SPARK-44462:
---------------------------------
    Priority: Critical  (was: Major)

> Fix the session passed to foreachBatch. 
> ----------------------------------------
>
>                 Key: SPARK-44462
>                 URL: https://issues.apache.org/jira/browse/SPARK-44462
>             Project: Spark
>          Issue Type: Task
>          Components: Connect, Structured Streaming
>    Affects Versions: 3.4.1
>            Reporter: Raghu Angadi
>            Priority: Critical
>
> foreachBatch() in Connect uses initial session used while starting the streaming query. But streaming query uses a cloned session, not the the original session. We should set up the mapping for the cloned session and pass that in. Look for this ticket ID in the code for more context inline. 
>  
> Another issue with not creating new session id: foreachBatch worker keeps the session alive. The session mapping at Connect server does not expire and query keeps running even if the original client disappears. This keeps the query running.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org