You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by SRK <sw...@gmail.com> on 2016/02/15 05:40:50 UTC
How to query a hive table from inside a map in Spark
Hi,
Is it possible to query a hive table which has data stored in the form of a
parquet file from inside map/partitions in Spark? My requirement is that I
have a User table in Hive/hdfs and for each record inside a sessions RDD, I
should be able to query the User table and if the User table already has a
record for that userId, query the record and do further processing.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-query-a-hive-table-from-inside-a-map-in-Spark-tp26224.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to query a hive table from inside a map in Spark
Posted by Alex Kozlov <al...@gmail.com>.
While this is possible via jdbc calls, it is not the best practice: you
should probably use variable broadcasting
<http://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables>
instead.
On Sun, Feb 14, 2016 at 8:40 PM, SRK <sw...@gmail.com> wrote:
> Hi,
>
> Is it possible to query a hive table which has data stored in the form of a
> parquet file from inside map/partitions in Spark? My requirement is that I
> have a User table in Hive/hdfs and for each record inside a sessions RDD, I
> should be able to query the User table and if the User table already has a
> record for that userId, query the record and do further processing.
>
>
> Thanks!
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-query-a-hive-table-from-inside-a-map-in-Spark-tp26224.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com