You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ilayaperumal Gopinathan (JIRA)" <ji...@apache.org> on 2014/12/10 01:54:15 UTC
[jira] [Commented] (SPARK-2892) Socket Receiver does not stop when
streaming context is stopped
[ https://issues.apache.org/jira/browse/SPARK-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240437#comment-14240437 ]
Ilayaperumal Gopinathan commented on SPARK-2892:
------------------------------------------------
It looks like this one and the issue mentioned in SPARK-4802 (ReceiverInfo removal at ReceiverTracker upon deregistering receiver) are related. I believe the following warning message is the result of receiverInfo not being removed at ReceiverTracker by the ReceiverTrackerActor when the corresponding receiver is deregistered.
"WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,SocketReceiver-0,null,false,localhost,Stopped by driver,))"
>From what I can see so far, closing the streaming context stops the receiver only in "local" mode.
In "cluster" mode, using the Spark standalone cluster I noticed that when the ReceiverTracker at the driver sends the "StopReceiver" message as a result of streaming context close, it couldn't reach to the ReceiverSupervisorImpl's actor that is running at the executor node. At the same time, the ReceiverSupervisorImpl at the executor could send the messages such as RegisterReceiver, AddBlock back to the ReceiverTrackerActor at the driver.
It would be great if someone could explain what might be going on from ReceiverTracker -> ReceiverSupervisorImpl actor at executor when sending the stop signal in the distributed mode case.
Thanks!
> Socket Receiver does not stop when streaming context is stopped
> ---------------------------------------------------------------
>
> Key: SPARK-2892
> URL: https://issues.apache.org/jira/browse/SPARK-2892
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.0.2
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Priority: Critical
>
> Running NetworkWordCount with
> {quote}
> ssc.start(); Thread.sleep(10000); ssc.stop(stopSparkContext = false); Thread.sleep(60000)
> {quote}
> gives the following error
> {quote}
> 14/08/06 18:37:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 10047 ms on localhost (1/1)
> 14/08/06 18:37:13 INFO DAGScheduler: Stage 0 (runJob at ReceiverTracker.scala:275) finished in 10.056 s
> 14/08/06 18:37:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
> 14/08/06 18:37:13 INFO SparkContext: Job finished: runJob at ReceiverTracker.scala:275, took 10.179263 s
> 14/08/06 18:37:13 INFO ReceiverTracker: All of the receivers have been terminated
> 14/08/06 18:37:13 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,SocketReceiver-0,null,false,localhost,Stopped by driver,))
> 14/08/06 18:37:13 INFO ReceiverTracker: ReceiverTracker stopped
> 14/08/06 18:37:13 INFO JobGenerator: Stopping JobGenerator immediately
> 14/08/06 18:37:13 INFO RecurringTimer: Stopped timer for JobGenerator after time 1407375433000
> 14/08/06 18:37:13 INFO JobGenerator: Stopped JobGenerator
> 14/08/06 18:37:13 INFO JobScheduler: Stopped JobScheduler
> 14/08/06 18:37:13 INFO StreamingContext: StreamingContext stopped successfully
> 14/08/06 18:37:43 INFO SocketReceiver: Stopped receiving
> 14/08/06 18:37:43 INFO SocketReceiver: Closed socket to localhost:9999
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org