You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Christian Tzolov <ct...@pivotal.io> on 2015/08/28 16:53:13 UTC
[DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
I've been exploring lately how would a streaming platforms fit into
Zeppelin.
The streaming processes are continuous as opposed to the usual Paragraph's
batch like jobs.
For example you start the twitter search stream in a paragraph (by clicking
the Run button) the underlying streaming platform will open a stream and
will return and the Paragraph will show a 'FINISHED' state while the real
state in this case is 'STARTED' (or alike)?
Currently Zeppelin doesn't seem to provide abstractions to track and stop
such long-living tasks?
I've checked the Spark Streaming tutorial but couldn't find an explanation
of how to track and stop running streams?
Do you think this is a valid use case for Zeppelin? Perhaps Spark Streaming
and Flink have already solved this somehow?
If you think this is valid use case i will open a JIRA ticket so we can
start looking for solutions.
Cheers,
Christian
--
Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
EMEA Practice Team | Pivotal <http://pivotal.io/>
ctzolov@pivotal.io|+31610285517
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by DuyHai Doan <do...@gmail.com>.
To be honest, I never succeeded to execute the twitter streaming example
with Zeppelin. Since the Scheduler is FIFO, once you launch the paragraph,
it stays forever in pending state so I couldn't stop the stream apart from
killing the zeppelin process.
So + 1 on the idea of streaming abstraction
On Fri, Aug 28, 2015 at 5:17 PM, Amos B. Elberg <am...@gmail.com>
wrote:
> Seems like we have two separate components. One is to allow interpreters
> to update interpreter results. The other is interpreters that do that.
>
> > On Aug 28, 2015, at 11:02 AM, IT CTO <go...@gmail.com> wrote:
> >
> > +1 for streaming support.
> >
> > בתאריך יום ו׳, 28 באוג׳ 2015, 17:53 מאת Christian Tzolov <
> ctzolov@pivotal.io
> >> :
> >
> >> I've been exploring lately how would a streaming platforms fit into
> >> Zeppelin.
> >>
> >> The streaming processes are continuous as opposed to the usual
> Paragraph's
> >> batch like jobs.
> >>
> >> For example you start the twitter search stream in a paragraph (by
> clicking
> >> the Run button) the underlying streaming platform will open a stream and
> >> will return and the Paragraph will show a 'FINISHED' state while the
> real
> >> state in this case is 'STARTED' (or alike)?
> >>
> >> Currently Zeppelin doesn't seem to provide abstractions to track and
> stop
> >> such long-living tasks?
> >>
> >> I've checked the Spark Streaming tutorial but couldn't find an
> explanation
> >> of how to track and stop running streams?
> >>
> >> Do you think this is a valid use case for Zeppelin? Perhaps Spark
> Streaming
> >> and Flink have already solved this somehow?
> >>
> >> If you think this is valid use case i will open a JIRA ticket so we can
> >> start looking for solutions.
> >>
> >> Cheers,
> >> Christian
> >>
> >> --
> >> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution
> Architect,
> >> EMEA Practice Team | Pivotal <http://pivotal.io/>
> >> ctzolov@pivotal.io|+31610285517
> >>
>
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by "Amos B. Elberg" <am...@gmail.com>.
Seems like we have two separate components. One is to allow interpreters to update interpreter results. The other is interpreters that do that.
> On Aug 28, 2015, at 11:02 AM, IT CTO <go...@gmail.com> wrote:
>
> +1 for streaming support.
>
> בתאריך יום ו׳, 28 באוג׳ 2015, 17:53 מאת Christian Tzolov <ctzolov@pivotal.io
>> :
>
>> I've been exploring lately how would a streaming platforms fit into
>> Zeppelin.
>>
>> The streaming processes are continuous as opposed to the usual Paragraph's
>> batch like jobs.
>>
>> For example you start the twitter search stream in a paragraph (by clicking
>> the Run button) the underlying streaming platform will open a stream and
>> will return and the Paragraph will show a 'FINISHED' state while the real
>> state in this case is 'STARTED' (or alike)?
>>
>> Currently Zeppelin doesn't seem to provide abstractions to track and stop
>> such long-living tasks?
>>
>> I've checked the Spark Streaming tutorial but couldn't find an explanation
>> of how to track and stop running streams?
>>
>> Do you think this is a valid use case for Zeppelin? Perhaps Spark Streaming
>> and Flink have already solved this somehow?
>>
>> If you think this is valid use case i will open a JIRA ticket so we can
>> start looking for solutions.
>>
>> Cheers,
>> Christian
>>
>> --
>> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
>> EMEA Practice Team | Pivotal <http://pivotal.io/>
>> ctzolov@pivotal.io|+31610285517
>>
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by IT CTO <go...@gmail.com>.
+1 for streaming support.
בתאריך יום ו׳, 28 באוג׳ 2015, 17:53 מאת Christian Tzolov <ctzolov@pivotal.io
>:
> I've been exploring lately how would a streaming platforms fit into
> Zeppelin.
>
> The streaming processes are continuous as opposed to the usual Paragraph's
> batch like jobs.
>
> For example you start the twitter search stream in a paragraph (by clicking
> the Run button) the underlying streaming platform will open a stream and
> will return and the Paragraph will show a 'FINISHED' state while the real
> state in this case is 'STARTED' (or alike)?
>
> Currently Zeppelin doesn't seem to provide abstractions to track and stop
> such long-living tasks?
>
> I've checked the Spark Streaming tutorial but couldn't find an explanation
> of how to track and stop running streams?
>
> Do you think this is a valid use case for Zeppelin? Perhaps Spark Streaming
> and Flink have already solved this somehow?
>
> If you think this is valid use case i will open a JIRA ticket so we can
> start looking for solutions.
>
> Cheers,
> Christian
>
> --
> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
> EMEA Practice Team | Pivotal <http://pivotal.io/>
> ctzolov@pivotal.io|+31610285517
>
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by Henry Saputra <he...@gmail.com>.
Hi @Christian, sorry about the late response. Will follow up in the
JIRA ticket for discussions.
- Henry
On Tue, Sep 1, 2015 at 3:44 PM, Christian Tzolov <ct...@pivotal.io> wrote:
> just opened https://issues.apache.org/jira/browse/ZEPPELIN-274 to track
> this proposal.
>
> @Henry, IMO the issue is not related to the underlying streaming platforms
> which provides various means for tracking the status as it is with the
> Zeppelin lifecycle model which doesn't provide abstractions to support
> long-running jobs. See the JIRA ticket for additional info.
>
> On 30 August 2015 at 03:17, Henry Saputra <he...@gmail.com> wrote:
>
>> So what is need from any underlying streaming framework to support
>> this on Zeppelin?
>> I guess something like state updates for each record processed?
>>
>> On Fri, Aug 28, 2015 at 7:53 AM, Christian Tzolov <ct...@pivotal.io>
>> wrote:
>> > I've been exploring lately how would a streaming platforms fit into
>> > Zeppelin.
>> >
>> > The streaming processes are continuous as opposed to the usual
>> Paragraph's
>> > batch like jobs.
>> >
>> > For example you start the twitter search stream in a paragraph (by
>> clicking
>> > the Run button) the underlying streaming platform will open a stream and
>> > will return and the Paragraph will show a 'FINISHED' state while the real
>> > state in this case is 'STARTED' (or alike)?
>> >
>> > Currently Zeppelin doesn't seem to provide abstractions to track and stop
>> > such long-living tasks?
>> >
>> > I've checked the Spark Streaming tutorial but couldn't find an
>> explanation
>> > of how to track and stop running streams?
>> >
>> > Do you think this is a valid use case for Zeppelin? Perhaps Spark
>> Streaming
>> > and Flink have already solved this somehow?
>> >
>> > If you think this is valid use case i will open a JIRA ticket so we can
>> > start looking for solutions.
>> >
>> > Cheers,
>> > Christian
>> >
>> > --
>> > Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution
>> Architect,
>> > EMEA Practice Team | Pivotal <http://pivotal.io/>
>> > ctzolov@pivotal.io|+31610285517
>>
>
>
>
> --
> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
> EMEA Practice Team | Pivotal <http://pivotal.io/>
> ctzolov@pivotal.io|+31610285517
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by Christian Tzolov <ct...@pivotal.io>.
just opened https://issues.apache.org/jira/browse/ZEPPELIN-274 to track
this proposal.
@Henry, IMO the issue is not related to the underlying streaming platforms
which provides various means for tracking the status as it is with the
Zeppelin lifecycle model which doesn't provide abstractions to support
long-running jobs. See the JIRA ticket for additional info.
On 30 August 2015 at 03:17, Henry Saputra <he...@gmail.com> wrote:
> So what is need from any underlying streaming framework to support
> this on Zeppelin?
> I guess something like state updates for each record processed?
>
> On Fri, Aug 28, 2015 at 7:53 AM, Christian Tzolov <ct...@pivotal.io>
> wrote:
> > I've been exploring lately how would a streaming platforms fit into
> > Zeppelin.
> >
> > The streaming processes are continuous as opposed to the usual
> Paragraph's
> > batch like jobs.
> >
> > For example you start the twitter search stream in a paragraph (by
> clicking
> > the Run button) the underlying streaming platform will open a stream and
> > will return and the Paragraph will show a 'FINISHED' state while the real
> > state in this case is 'STARTED' (or alike)?
> >
> > Currently Zeppelin doesn't seem to provide abstractions to track and stop
> > such long-living tasks?
> >
> > I've checked the Spark Streaming tutorial but couldn't find an
> explanation
> > of how to track and stop running streams?
> >
> > Do you think this is a valid use case for Zeppelin? Perhaps Spark
> Streaming
> > and Flink have already solved this somehow?
> >
> > If you think this is valid use case i will open a JIRA ticket so we can
> > start looking for solutions.
> >
> > Cheers,
> > Christian
> >
> > --
> > Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution
> Architect,
> > EMEA Practice Team | Pivotal <http://pivotal.io/>
> > ctzolov@pivotal.io|+31610285517
>
--
Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
EMEA Practice Team | Pivotal <http://pivotal.io/>
ctzolov@pivotal.io|+31610285517
Re: [DISCUSSION] Tracking Streaming and Long-living Tasks in Zeppelin
Posted by Henry Saputra <he...@gmail.com>.
So what is need from any underlying streaming framework to support
this on Zeppelin?
I guess something like state updates for each record processed?
On Fri, Aug 28, 2015 at 7:53 AM, Christian Tzolov <ct...@pivotal.io> wrote:
> I've been exploring lately how would a streaming platforms fit into
> Zeppelin.
>
> The streaming processes are continuous as opposed to the usual Paragraph's
> batch like jobs.
>
> For example you start the twitter search stream in a paragraph (by clicking
> the Run button) the underlying streaming platform will open a stream and
> will return and the Paragraph will show a 'FINISHED' state while the real
> state in this case is 'STARTED' (or alike)?
>
> Currently Zeppelin doesn't seem to provide abstractions to track and stop
> such long-living tasks?
>
> I've checked the Spark Streaming tutorial but couldn't find an explanation
> of how to track and stop running streams?
>
> Do you think this is a valid use case for Zeppelin? Perhaps Spark Streaming
> and Flink have already solved this somehow?
>
> If you think this is valid use case i will open a JIRA ticket so we can
> start looking for solutions.
>
> Cheers,
> Christian
>
> --
> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution Architect,
> EMEA Practice Team | Pivotal <http://pivotal.io/>
> ctzolov@pivotal.io|+31610285517