You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2024/06/27 01:54:00 UTC
[jira] [Assigned] (SPARK-48717) Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries
[ https://issues.apache.org/jira/browse/SPARK-48717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-48717:
------------------------------------
Assignee: Wei Liu
> Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48717
> URL: https://issues.apache.org/jira/browse/SPARK-48717
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, SS
> Affects Versions: 4.0.0
> Reporter: Wei Liu
> Assignee: Wei Liu
> Priority: Major
>
> Followup of https://issues.apache.org/jira/browse/SPARK-39218
>
> It only considered the InterruptedException is thrown when time.sleep(10) is intercepted. But when a spark query is executing:
> {code:java}
> def func(batch_df, batch_id):
> batch_df.sparkSession.range(10000000).write.saveAsTable("oops")
> print(batch_df.count()) {code}
> the actual error would be:
>
> {code:java}
> py4j.protocol.Py4JJavaError: An error occurred while calling o2141502.saveAsTable.
> : org.apache.spark.SparkException: Job aborted.
> <REDACTED>
> <REDACTED>
> <REDACTED>
> ...
> at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$2(StreamExecution.scala:262)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:262)
> *Caused by: java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1000)*
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308) {code}
> We should also add consideration to this scenario
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org