You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by SRK <sw...@gmail.com> on 2015/12/23 01:44:34 UTC
Can SqlContext be used inside mapPartitions
Hi,
Can SQL Context be used inside mapPartitions? My requirement is to register
a set of data from hdfs as a temp table and to be able to lookup from inside
MapPartitions based on a key. If it is not supported, is there a different
way of doing this?
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-SqlContext-be-used-inside-mapPartitions-tp25771.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Can SqlContext be used inside mapPartitions
Posted by Ted Yu <yu...@gmail.com>.
bq. be able to lookup from inside MapPartitions based on a key
Please describe your use case in bit more detail.
One possibility is to use NoSQL database such as HBase. There're several
choices for Spark HBase connector.
Cheers
On Tue, Dec 22, 2015 at 4:51 PM, Zhan Zhang <zz...@hortonworks.com> wrote:
> SQLContext is in driver side, and I don’t think you can use it in
> executors.
> How to provide lookup functionality in executors really depends on how you
> would use them.
>
> Thanks.
>
> Zhan Zhang
>
> On Dec 22, 2015, at 4:44 PM, SRK <sw...@gmail.com> wrote:
>
> > Hi,
> >
> > Can SQL Context be used inside mapPartitions? My requirement is to
> register
> > a set of data from hdfs as a temp table and to be able to lookup from
> inside
> > MapPartitions based on a key. If it is not supported, is there a
> different
> > way of doing this?
> >
> > Thanks!
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-SqlContext-be-used-inside-mapPartitions-tp25771.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Can SqlContext be used inside mapPartitions
Posted by Zhan Zhang <zz...@hortonworks.com>.
SQLContext is in driver side, and I don’t think you can use it in executors.
How to provide lookup functionality in executors really depends on how you would use them.
Thanks.
Zhan Zhang
On Dec 22, 2015, at 4:44 PM, SRK <sw...@gmail.com> wrote:
> Hi,
>
> Can SQL Context be used inside mapPartitions? My requirement is to register
> a set of data from hdfs as a temp table and to be able to lookup from inside
> MapPartitions based on a key. If it is not supported, is there a different
> way of doing this?
>
> Thanks!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-SqlContext-be-used-inside-mapPartitions-tp25771.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org