You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jamborta <ja...@gmail.com> on 2015/02/13 11:04:20 UTC

Interact with streams in a non-blocking way

Hi all,

I am trying to come up with a workflow where I can query streams
asynchronously. The problem I have is a ssc.awaitTermination() line blocks
the whole thread, so it is not straightforward to me whether it is possible
to get hold of objects from streams once they are started. any suggestion on
what is the best way to implement this?

thanks,



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Interact-with-streams-in-a-non-blocking-way-tp21640.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Interact with streams in a non-blocking way

Posted by Tathagata Das <td...@databricks.com>.
Here is an example of how you can do. Lets say "myDStream" contains the
data that you may want to asynchornously query, say using, Spark SQL.

val sqlContext = new SqlContext(streamingContext.sparkContext)

myDStream.foreachRDD { rdd =>   // rdd is a RDD of case class
   sqlContext.registerRDDAsTable(rdd, "mytable")
}

This will make sure that the table "mytable" always refers to the latest
RDD generated by the DStream.
Then from a diffrent thread you can asynchronously query

sqlContext.sql("select * from mytable")

Hope this helps.

TD



On Fri, Feb 13, 2015 at 3:59 AM, Sean Owen <so...@cloudera.com> wrote:

> Sure it's possible, but you would use Streaming to update some shared
> state, and create another service that accessed that shared state too.
>
> On Fri, Feb 13, 2015 at 11:57 AM, Tamas Jambor <ja...@gmail.com> wrote:
> > Thanks for the reply, I am trying to setup a streaming as a service
> > approach, using the framework that is used for spark-jobserver. for that
> I
> > would need to handle asynchronous  operations that are initiated from
> > outside of the stream. Do you think it is not possible?
> >
> > On Fri Feb 13 2015 at 10:14:18 Sean Owen <so...@cloudera.com> wrote:
> >>
> >> You call awaitTermination() in the main thread, and indeed it blocks
> >> there forever. From there Spark Streaming takes over, and is invoking
> >> the operations you set up. Your operations have access to the data of
> >> course. That's the model; you don't make external threads that reach
> >> in to Spark Streaming's objects, but can easily create operations that
> >> take whatever actions you want and invoke them in Streaming.
> >>
> >> On Fri, Feb 13, 2015 at 10:04 AM, jamborta <ja...@gmail.com> wrote:
> >> > Hi all,
> >> >
> >> > I am trying to come up with a workflow where I can query streams
> >> > asynchronously. The problem I have is a ssc.awaitTermination() line
> >> > blocks
> >> > the whole thread, so it is not straightforward to me whether it is
> >> > possible
> >> > to get hold of objects from streams once they are started. any
> >> > suggestion on
> >> > what is the best way to implement this?
> >> >
> >> > thanks,
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://apache-spark-user-list.1001560.n3.nabble.com/Interact-with-streams-in-a-non-blocking-way-tp21640.html
> >> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> > For additional commands, e-mail: user-help@spark.apache.org
> >> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Interact with streams in a non-blocking way

Posted by Sean Owen <so...@cloudera.com>.
Sure it's possible, but you would use Streaming to update some shared
state, and create another service that accessed that shared state too.

On Fri, Feb 13, 2015 at 11:57 AM, Tamas Jambor <ja...@gmail.com> wrote:
> Thanks for the reply, I am trying to setup a streaming as a service
> approach, using the framework that is used for spark-jobserver. for that I
> would need to handle asynchronous  operations that are initiated from
> outside of the stream. Do you think it is not possible?
>
> On Fri Feb 13 2015 at 10:14:18 Sean Owen <so...@cloudera.com> wrote:
>>
>> You call awaitTermination() in the main thread, and indeed it blocks
>> there forever. From there Spark Streaming takes over, and is invoking
>> the operations you set up. Your operations have access to the data of
>> course. That's the model; you don't make external threads that reach
>> in to Spark Streaming's objects, but can easily create operations that
>> take whatever actions you want and invoke them in Streaming.
>>
>> On Fri, Feb 13, 2015 at 10:04 AM, jamborta <ja...@gmail.com> wrote:
>> > Hi all,
>> >
>> > I am trying to come up with a workflow where I can query streams
>> > asynchronously. The problem I have is a ssc.awaitTermination() line
>> > blocks
>> > the whole thread, so it is not straightforward to me whether it is
>> > possible
>> > to get hold of objects from streams once they are started. any
>> > suggestion on
>> > what is the best way to implement this?
>> >
>> > thanks,
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/Interact-with-streams-in-a-non-blocking-way-tp21640.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Interact with streams in a non-blocking way

Posted by Tamas Jambor <ja...@gmail.com>.
Thanks for the reply, I am trying to setup a streaming as a service
approach, using the framework that is used for spark-jobserver. for that I
would need to handle asynchronous  operations that are initiated from
outside of the stream. Do you think it is not possible?

On Fri Feb 13 2015 at 10:14:18 Sean Owen <so...@cloudera.com> wrote:

> You call awaitTermination() in the main thread, and indeed it blocks
> there forever. From there Spark Streaming takes over, and is invoking
> the operations you set up. Your operations have access to the data of
> course. That's the model; you don't make external threads that reach
> in to Spark Streaming's objects, but can easily create operations that
> take whatever actions you want and invoke them in Streaming.
>
> On Fri, Feb 13, 2015 at 10:04 AM, jamborta <ja...@gmail.com> wrote:
> > Hi all,
> >
> > I am trying to come up with a workflow where I can query streams
> > asynchronously. The problem I have is a ssc.awaitTermination() line
> blocks
> > the whole thread, so it is not straightforward to me whether it is
> possible
> > to get hold of objects from streams once they are started. any
> suggestion on
> > what is the best way to implement this?
> >
> > thanks,
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Interact-with-streams-in-a-
> non-blocking-way-tp21640.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>

Re: Interact with streams in a non-blocking way

Posted by Sean Owen <so...@cloudera.com>.
You call awaitTermination() in the main thread, and indeed it blocks
there forever. From there Spark Streaming takes over, and is invoking
the operations you set up. Your operations have access to the data of
course. That's the model; you don't make external threads that reach
in to Spark Streaming's objects, but can easily create operations that
take whatever actions you want and invoke them in Streaming.

On Fri, Feb 13, 2015 at 10:04 AM, jamborta <ja...@gmail.com> wrote:
> Hi all,
>
> I am trying to come up with a workflow where I can query streams
> asynchronously. The problem I have is a ssc.awaitTermination() line blocks
> the whole thread, so it is not straightforward to me whether it is possible
> to get hold of objects from streams once they are started. any suggestion on
> what is the best way to implement this?
>
> thanks,
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Interact-with-streams-in-a-non-blocking-way-tp21640.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org