You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sranga <sr...@gmail.com> on 2014/12/30 03:45:50 UTC
Spark Streaming: HiveContext within Custom Actor
Hi
Could Spark-SQL be used from within a custom actor that acts as a receiver
for a streaming application? If yes, what is the recommended way of passing
the SparkContext to the actor?
Thanks for your help.
- Ranga
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-HiveContext-within-Custom-Actor-tp20892.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming: HiveContext within Custom Actor
Posted by Ranga <sr...@gmail.com>.
Thanks. Will look at other options.
On Tue, Dec 30, 2014 at 11:43 AM, Tathagata Das <tathagata.das1565@gmail.com
> wrote:
> I am not sure that can be done. Receivers are designed to be run only
> on the executors/workers, whereas a SQLContext (for using Spark SQL)
> can only be defined on the driver.
>
>
> On Mon, Dec 29, 2014 at 6:45 PM, sranga <sr...@gmail.com> wrote:
> > Hi
> >
> > Could Spark-SQL be used from within a custom actor that acts as a
> receiver
> > for a streaming application? If yes, what is the recommended way of
> passing
> > the SparkContext to the actor?
> > Thanks for your help.
> >
> >
> > - Ranga
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-HiveContext-within-Custom-Actor-tp20892.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: Spark Streaming: HiveContext within Custom Actor
Posted by Tathagata Das <ta...@gmail.com>.
I am not sure that can be done. Receivers are designed to be run only
on the executors/workers, whereas a SQLContext (for using Spark SQL)
can only be defined on the driver.
On Mon, Dec 29, 2014 at 6:45 PM, sranga <sr...@gmail.com> wrote:
> Hi
>
> Could Spark-SQL be used from within a custom actor that acts as a receiver
> for a streaming application? If yes, what is the recommended way of passing
> the SparkContext to the actor?
> Thanks for your help.
>
>
> - Ranga
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-HiveContext-within-Custom-Actor-tp20892.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org