You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Deenar Toraskar <de...@db.com> on 2013/12/09 14:46:13 UTC
Spark API - support for asynchronous calls - Reactive style [I]
Classification: For internal use only
Hi developers
Are there any plans to have Spark (and Shark) APIs that are asynchronous and non blocking? APIs that return Futures and Iteratee/Enumerators would be very useful to users building scalable apps using Spark, specially when combined with a fully asynchronous/non-blocking framework like Play!.
Something along the lines of ReactiveMongo
http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
Deenar
---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Please refer to http://www.db.com/en/content/eu_disclosures.htm for additional EU corporate and regulatory disclosures.
Re: Spark API - support for asynchronous calls - Reactive style [I]
Posted by Mark Hamstra <ma...@clearstorydata.com>.
I'm having fun with it. And it's almost all Reynold's work, so I can't
take credit for it.
On Thu, Dec 12, 2013 at 8:56 AM, Evan Chan <ev...@ooyala.com> wrote:
> Mark,
>
> Thanks. The FutureAction API looks awesome.
>
>
> On Mon, Dec 9, 2013 at 9:31 AM, Mark Hamstra <mark@clearstorydata.com
> >wrote:
>
> > Spark has already supported async jobs for awhile now --
> > https://github.com/apache/incubator-spark/pull/29, and they even work
> > correctly after https://github.com/apache/incubator-spark/pull/232
> >
> > There are now implicit conversions from RDD to
> > AsyncRDDActions<
> >
> https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala
> > >,
> > where async actions like countAsync are defined.
> >
> >
> > On Mon, Dec 9, 2013 at 5:46 AM, Deenar Toraskar <deenar.toraskar@db.com
> > >wrote:
> >
> > > Classification: For internal use only
> > > Hi developers
> > >
> > > Are there any plans to have Spark (and Shark) APIs that are
> asynchronous
> > > and non blocking? APIs that return Futures and Iteratee/Enumerators
> would
> > > be very useful to users building scalable apps using Spark, specially
> > when
> > > combined with a fully asynchronous/non-blocking framework like Play!.
> > >
> > > Something along the lines of ReactiveMongo
> > >
> > >
> >
> http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
> > >
> > >
> > > Deenar
> > >
> > > ---
> > > This e-mail may contain confidential and/or privileged information. If
> > you
> > > are not the intended recipient (or have received this e-mail in error)
> > > please notify the sender immediately and delete this e-mail. Any
> > > unauthorized copying, disclosure or distribution of the material in
> this
> > > e-mail is strictly forbidden.
> > >
> > > Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> > > additional EU corporate and regulatory disclosures.
> > >
> >
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
> http://www.twitter.com/ooyala>
>
Re: Spark API - support for asynchronous calls - Reactive style [I]
Posted by Evan Chan <ev...@ooyala.com>.
Mark,
Thanks. The FutureAction API looks awesome.
On Mon, Dec 9, 2013 at 9:31 AM, Mark Hamstra <ma...@clearstorydata.com>wrote:
> Spark has already supported async jobs for awhile now --
> https://github.com/apache/incubator-spark/pull/29, and they even work
> correctly after https://github.com/apache/incubator-spark/pull/232
>
> There are now implicit conversions from RDD to
> AsyncRDDActions<
> https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala
> >,
> where async actions like countAsync are defined.
>
>
> On Mon, Dec 9, 2013 at 5:46 AM, Deenar Toraskar <deenar.toraskar@db.com
> >wrote:
>
> > Classification: For internal use only
> > Hi developers
> >
> > Are there any plans to have Spark (and Shark) APIs that are asynchronous
> > and non blocking? APIs that return Futures and Iteratee/Enumerators would
> > be very useful to users building scalable apps using Spark, specially
> when
> > combined with a fully asynchronous/non-blocking framework like Play!.
> >
> > Something along the lines of ReactiveMongo
> >
> >
> http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
> >
> >
> > Deenar
> >
> > ---
> > This e-mail may contain confidential and/or privileged information. If
> you
> > are not the intended recipient (or have received this e-mail in error)
> > please notify the sender immediately and delete this e-mail. Any
> > unauthorized copying, disclosure or distribution of the material in this
> > e-mail is strictly forbidden.
> >
> > Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> > additional EU corporate and regulatory disclosures.
> >
>
--
--
Evan Chan
Staff Engineer
ev@ooyala.com |
<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: Spark API - support for asynchronous calls - Reactive style [I]
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Spark has already supported async jobs for awhile now --
https://github.com/apache/incubator-spark/pull/29, and they even work
correctly after https://github.com/apache/incubator-spark/pull/232
There are now implicit conversions from RDD to
AsyncRDDActions<https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala>,
where async actions like countAsync are defined.
On Mon, Dec 9, 2013 at 5:46 AM, Deenar Toraskar <de...@db.com>wrote:
> Classification: For internal use only
> Hi developers
>
> Are there any plans to have Spark (and Shark) APIs that are asynchronous
> and non blocking? APIs that return Futures and Iteratee/Enumerators would
> be very useful to users building scalable apps using Spark, specially when
> combined with a fully asynchronous/non-blocking framework like Play!.
>
> Something along the lines of ReactiveMongo
>
> http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
>
>
> Deenar
>
> ---
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and delete this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
> Please refer to http://www.db.com/en/content/eu_disclosures.htm for
> additional EU corporate and regulatory disclosures.
>