Genmao Yu created SPARK-27503:
---------------------------------
Summary: JobGenerator thread exit for some fatal errors but application keeps running
Key: SPARK-27503
URL: https://issues.apache.org/jira/browse/SPARK-27503
Project: Spark
Issue Type: Bug
Components: DStreams
Affects Versions: 3.0.0
Reporter: Genmao Yu
JobGenerator thread (including some other EventLoop threads) may exit for some fatal error, like OOM, but Spark Streaming job keep running with no batch job generating. Currently, we only report any non-fatal error.
{code}
override def run(): Unit = {
try {
while (!stopped.get) {
val event = eventQueue.take()
try {
onReceive(event)
} catch {
case NonFatal(e) =>
try {
onError(e)
} catch {
case NonFatal(e) => logError("Unexpected error in " + name, e)
}
}
}
} catch {
case ie: InterruptedException => // exit even if eventQueue is not empty
case NonFatal(e) => logError("Unexpected error in " + name, e)
}
}
{code}
In some corner cases, these event threads may exit with OOM error, but driver thread can still keep running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org