You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nick Chammas <ni...@gmail.com> on 2014/07/13 00:03:39 UTC
Stopping StreamingContext does not kill receiver
>From the interactive shell I’ve created a StreamingContext.
I call ssc.start() and take a look at http://master_url:4040/streaming/ and
see that I have an active Twitter receiver. Then I call
ssc.stop(stopSparkContext
= false, stopGracefully = true) and wait a bit, but the receiver seems to
stay active.
Is this expected? I’m running 1.0.1 on EC2.
Nick
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Stopping StreamingContext does not kill receiver
Posted by Tathagata Das <ta...@gmail.com>.
I narrowed down the error. Unfortunately this is not quick fix. I have
opened a JIRA for this.
https://issues.apache.org/jira/browse/SPARK-2892
On Wed, Aug 6, 2014 at 3:59 PM, Tathagata Das <ta...@gmail.com>
wrote:
> Okay let me give it a shot.
>
>
> On Wed, Aug 6, 2014 at 3:57 PM, lbustelo <gi...@bustelos.com> wrote:
>
>> Sorry about the screenshot… but that is what I have handy right now. You
>> can
>> see that we get a WARN and it ultimately say that it stopped successfully.
>> When looking that the application in Spark UI, it still shows the
>> ReceiverTracker task running.
>>
>> It is easy to recreate. On the spark repl we are running a modified
>> version
>> of
>>
>> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala
>> .
>>
>> Then do a ssc.stop(false).
>>
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png
>> >
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Stopping StreamingContext does not kill receiver
Posted by Tathagata Das <ta...@gmail.com>.
Okay let me give it a shot.
On Wed, Aug 6, 2014 at 3:57 PM, lbustelo <gi...@bustelos.com> wrote:
> Sorry about the screenshot… but that is what I have handy right now. You
> can
> see that we get a WARN and it ultimately say that it stopped successfully.
> When looking that the application in Spark UI, it still shows the
> ReceiverTracker task running.
>
> It is easy to recreate. On the spark repl we are running a modified version
> of
>
> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala
> .
>
> Then do a ssc.stop(false).
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Stopping StreamingContext does not kill receiver
Posted by lbustelo <gi...@bustelos.com>.
Sorry about the screenshot… but that is what I have handy right now. You can
see that we get a WARN and it ultimately say that it stopped successfully.
When looking that the application in Spark UI, it still shows the
ReceiverTracker task running.
It is easy to recreate. On the spark repl we are running a modified version
of
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala.
Then do a ssc.stop(false).
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Stopping StreamingContext does not kill receiver
Posted by Tathagata Das <ta...@gmail.com>.
Can you give the stack trace?
This was the fix for the twitter stream.
https://github.com/apache/spark/pull/1577/files
You could try doing the same.
TD
On Wed, Aug 6, 2014 at 2:41 PM, lbustelo <gi...@bustelos.com> wrote:
> I'm running on spark 1.0.0 and I see a similar problem when using the
> socketTextStream receiver. The ReceiverTracker task sticks around after a
> ssc.stop(false).
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Stopping StreamingContext does not kill receiver
Posted by lbustelo <gi...@bustelos.com>.
I'm running on spark 1.0.0 and I see a similar problem when using the
socketTextStream receiver. The ReceiverTracker task sticks around after a
ssc.stop(false).
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Stopping StreamingContext does not kill receiver
Posted by Nicholas Chammas <ni...@gmail.com>.
Okie doke. Thanks for filing the JIRA.
On Sat, Jul 12, 2014 at 6:45 PM, Tathagata Das <ta...@gmail.com>
wrote:
> Yes, thats a bug i just discovered. Race condition in the Twitter
> Receiver, will fix asap.
> Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464
>
> TD
>
>
> On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> To add a potentially relevant piece of information, around when I stop
>> the StreamingContext, I get the following warning:
>>
>> 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))
>>
>> Nick
>>
>>
>>
>> On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <nicholas.chammas@gmail.com
>> > wrote:
>>
>>> From the interactive shell I’ve created a StreamingContext.
>>>
>>> I call ssc.start() and take a look at http://master_url:4040/streaming/
>>> and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext
>>> = false, stopGracefully = true) and wait a bit, but the receiver seems
>>> to stay active.
>>>
>>> Is this expected? I’m running 1.0.1 on EC2.
>>>
>>> Nick
>>>
>>>
>>> ------------------------------
>>> View this message in context: Stopping StreamingContext does not kill
>>> receiver
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html>
>>> Sent from the Apache Spark User List mailing list archive
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>
>>
>>
>
Re: Stopping StreamingContext does not kill receiver
Posted by Tathagata Das <ta...@gmail.com>.
Yes, thats a bug i just discovered. Race condition in the Twitter Receiver,
will fix asap.
Here is the JIRA https://issues.apache.org/jira/browse/SPARK-2464
TD
On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> To add a potentially relevant piece of information, around when I stop the
> StreamingContext, I get the following warning:
>
> 14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))
>
> Nick
>
>
>
> On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <ni...@gmail.com>
> wrote:
>
>> From the interactive shell I’ve created a StreamingContext.
>>
>> I call ssc.start() and take a look at http://master_url:4040/streaming/
>> and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext
>> = false, stopGracefully = true) and wait a bit, but the receiver seems
>> to stay active.
>>
>> Is this expected? I’m running 1.0.1 on EC2.
>>
>> Nick
>>
>>
>> ------------------------------
>> View this message in context: Stopping StreamingContext does not kill
>> receiver
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
Re: Stopping StreamingContext does not kill receiver
Posted by Nicholas Chammas <ni...@gmail.com>.
To add a potentially relevant piece of information, around when I stop the
StreamingContext, I get the following warning:
14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not
deregistered, Map(0 ->
ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))
Nick
On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <ni...@gmail.com>
wrote:
> From the interactive shell I’ve created a StreamingContext.
>
> I call ssc.start() and take a look at http://master_url:4040/streaming/
> and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext
> = false, stopGracefully = true) and wait a bit, but the receiver seems to
> stay active.
>
> Is this expected? I’m running 1.0.1 on EC2.
>
> Nick
>
>
> ------------------------------
> View this message in context: Stopping StreamingContext does not kill
> receiver
> <http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>