You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Nikhil Utane <ni...@gmail.com> on 2017/07/05 05:19:36 UTC
Few issues and questions regarding streaming
Hi,
I am trying streaming with Zeppelin using the example given here: zepl.com
<https://www.zepl.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2hhZml6dXItcmFobWFuL2NjcC1jaGFsbGVuZ2UxL21hc3Rlci96ZXBwZWxpbi9ub3RlYm9vay8yQVdLWFhKTVgvbm90ZS5qc29u>
But not able to get it to work.
I have configured Spark interpreter with a master (spark://dk1:7077).
a) The first time it ran, the paragraph was in running state but it wasn't
displaying any output (sent using nc command).
Upon restarting it started giving error:
java.lang.IllegalStateException: Only one StreamingContext may be started
in this JVM. Currently running StreamingContext was started
atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:569)
I realized that I only have ssc.start() and there is no call to stop(). So
in another paragraph I tried calling ssc.stop() before re-running my main
paragraph but it is still not helping.
My question is, what is the right way to start/stop streaming app so that
we can run it multiple times back to back?
Is my expectation that I should be seeing output in the same window correct?
Pls let me know if any more info is needed.
-Thanks
Nikhil
Re: Few issues and questions regarding streaming
Posted by Nikhil Utane <ni...@gmail.com>.
Thanks for your response Arpit.
As per my understanding, only SparkContext is created. We have to create
the StreamingContext. The example that I am following is for Zeppelin where
StreamingContext is being created.
Now my problem is, I am getting a socket error while running my streaming
example. And I have to restart the interpreter every time I need to re-run
which is quite annoying. Hence I want to know graceful way so that I can
run the example multiple times without need for restarting the interpreter.
Even calling in this manner is not helping:
ssc.stop(stopSparkContext=false, stopGracefully=true)
-Thanks
Nikhil
On Wed, Jul 5, 2017 at 10:56 AM, arpit tak <ar...@gmail.com> wrote:
> Hi Nikhil,
> First, Spark context and streaming context already defined in zeppelin
> when you write code , you don’t need to initialize it again as we generally
> do in writing normal spark code.
> Second, you just need to start ssc.start() once and if your next
> paragraph is linked to first note then no need to call stop() you can do
> that at then end of last paragraph/note.
>
> Third, in case your note are independent of each other then at a time you
> can run only one of them .
>
> Hope this helps.
>
>
>
> On Jul 5, 2017, at 10:49 AM, Nikhil Utane <ni...@gmail.com>
> wrote:
>
> Hi,
>
> I am trying streaming with Zeppelin using the example given here: zepl.com
> <https://www.zepl.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2hhZml6dXItcmFobWFuL2NjcC1jaGFsbGVuZ2UxL21hc3Rlci96ZXBwZWxpbi9ub3RlYm9vay8yQVdLWFhKTVgvbm90ZS5qc29u>
> But not able to get it to work.
> I have configured Spark interpreter with a master (spark://dk1:7077).
>
> a) The first time it ran, the paragraph was in running state but it wasn't
> displaying any output (sent using nc command).
>
> Upon restarting it started giving error:
> java.lang.IllegalStateException: Only one StreamingContext may be started
> in this JVM. Currently running StreamingContext was started
> atorg.apache.spark.streaming.StreamingContext.start(
> StreamingContext.scala:569)
> I realized that I only have ssc.start() and there is no call to stop(). So
> in another paragraph I tried calling ssc.stop() before re-running my main
> paragraph but it is still not helping.
>
> My question is, what is the right way to start/stop streaming app so that
> we can run it multiple times back to back?
> Is my expectation that I should be seeing output in the same window
> correct?
>
> Pls let me know if any more info is needed.
>
> -Thanks
> Nikhil
>
>
>
Re: Few issues and questions regarding streaming
Posted by arpit tak <ar...@gmail.com>.
Hi Nikhil,
First, Spark context and streaming context already defined in zeppelin when you write code , you don’t need to initialize it again as we generally do in writing normal spark code.
Second, you just need to start ssc.start() once and if your next paragraph is linked to first note then no need to call stop() you can do that at then end of last paragraph/note.
Third, in case your note are independent of each other then at a time you can run only one of them .
Hope this helps.
> On Jul 5, 2017, at 10:49 AM, Nikhil Utane <ni...@gmail.com> wrote:
>
> Hi,
>
> I am trying streaming with Zeppelin using the example given here: zepl.com <https://www.zepl.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2hhZml6dXItcmFobWFuL2NjcC1jaGFsbGVuZ2UxL21hc3Rlci96ZXBwZWxpbi9ub3RlYm9vay8yQVdLWFhKTVgvbm90ZS5qc29u>
> But not able to get it to work.
> I have configured Spark interpreter with a master (spark://dk1:7077).
>
> a) The first time it ran, the paragraph was in running state but it wasn't displaying any output (sent using nc command).
>
> Upon restarting it started giving error:
> java.lang.IllegalStateException: Only one StreamingContext may be started in this JVM. Currently running StreamingContext was started atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:569)
> I realized that I only have ssc.start() and there is no call to stop(). So in another paragraph I tried calling ssc.stop() before re-running my main paragraph but it is still not helping.
>
> My question is, what is the right way to start/stop streaming app so that we can run it multiple times back to back?
> Is my expectation that I should be seeing output in the same window correct?
>
> Pls let me know if any more info is needed.
>
> -Thanks
> Nikhil
>