You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrei Taleanu (JIRA)" <ji...@apache.org> on 2017/04/13 11:45:41 UTC
[jira] [Updated] (SPARK-20321) Spark UI cannot be shutdown in spark
streaming app
[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Taleanu updated SPARK-20321:
-----------------------------------
Description:
When an exception thrown the transform stage is handled in foreachRDD and the streaming context is forced to stop, the SparkUI appears to hang and continually dump the following logs in an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
...
{noformat}
Unfortunately I don't have a minimal example that reproduces this issue but here is what I can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream.transform { rdd =>
val data = getData // Perform a call that potentially throws an exception
// broadcast the data
// flatMap the RDD using the data
}
mapped.foreachRDD {
try {
// write some data in a DB
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
mapped.foreachRDD {
try {
// write data to Kafka
// manually checkpoint the Kafka offsets (because I need them in JSON format)
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
{noformat}
The issue appears when stop is invoked. At the point when SparkUI is stopped, it enters that infinite loop. Initially I thought it relates to Jetty, as the version used in SparkUI had some bugs (e.g. [this one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a more recent version (March 2017) and built Spark 2.1.0 with that one but still got the error.
I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
was:
When an exception thrown the transform stage is handled in foreachRDD and the streaming context is forced to stop, the SparkUI appears to hang and continually dump the following logs in an infinite loop:
{noformat}
...
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
...
{noformat}
Unfortunately I don't have a minimal example that reproduces this issue but here is what I can share:
{noformat}
val dstream = pull data from kafka
val mapped = dstream transform { rdd =>
val data = getData // Perform a call that potentially throws an exception
// broadcast the data
// flatMap the RDD using the data
}
mapped.foreachRDD {
try {
// write some data in a DB
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
mapped.foreachRDD {
try {
// write data to Kafka
// manually checkpoint the Kafka offsets (because I need them in JSON format)
} catch {
case t: Throwable =>
dstream.context.stop(stopSparkContext = true, stopGracefully = false)
}
}
{noformat}
The issue appears when stop is invoked. At the point when SparkUI is stopped, it enters that infinite loop. Initially I thought it relates to Jetty, as the version used in SparkUI had some bugs (e.g. [this one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a more recent version (March 2017) but still got the error.
I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
> Spark UI cannot be shutdown in spark streaming app
> --------------------------------------------------
>
> Key: SPARK-20321
> URL: https://issues.apache.org/jira/browse/SPARK-20321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Andrei Taleanu
>
> When an exception thrown the transform stage is handled in foreachRDD and the streaming context is forced to stop, the SparkUI appears to hang and continually dump the following logs in an infinite loop:
> {noformat}
> ...
> 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
> 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
> 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop woken up from select, 0/0 selected
> 2017-04-12 14:11:13,470 [SparkUI-50-selector-ServerConnectorManager@512d4583/1] DEBUG org.spark_project.jetty.io.SelectorManager - Selector loop waiting on select
> ...
> {noformat}
> Unfortunately I don't have a minimal example that reproduces this issue but here is what I can share:
> {noformat}
> val dstream = pull data from kafka
> val mapped = dstream.transform { rdd =>
> val data = getData // Perform a call that potentially throws an exception
> // broadcast the data
> // flatMap the RDD using the data
> }
> mapped.foreachRDD {
> try {
> // write some data in a DB
> } catch {
> case t: Throwable =>
> dstream.context.stop(stopSparkContext = true, stopGracefully = false)
> }
> }
> mapped.foreachRDD {
> try {
> // write data to Kafka
> // manually checkpoint the Kafka offsets (because I need them in JSON format)
> } catch {
> case t: Throwable =>
> dstream.context.stop(stopSparkContext = true, stopGracefully = false)
> }
> }
> {noformat}
> The issue appears when stop is invoked. At the point when SparkUI is stopped, it enters that infinite loop. Initially I thought it relates to Jetty, as the version used in SparkUI had some bugs (e.g. [this one|https://bugs.eclipse.org/bugs/show_bug.cgi?id=452465]). I bumped Jetty to a more recent version (March 2017) and built Spark 2.1.0 with that one but still got the error.
> I encountered this issue with Spark 2.1.0 built with Hadoop 2.6 on top of Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org