You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nitinkak001 <ni...@gmail.com> on 2015/03/24 16:58:52 UTC
Does HiveContext connect to HiveServer2?
I am wondering if HiveContext connects to HiveServer2 or does it work though
Hive CLI. The reason I am asking is because Cloudera has deprecated Hive
CLI.
If the connection is through HiverServer2, is there a way to specify user
credentials?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Does HiveContext connect to HiveServer2?
Posted by Nitin kak <ni...@gmail.com>.
Hi Marcelo,
The issue does not happen while connecting to the hive metstore, that works
fine. It seems that HiveContext only uses Hive CLI to execute the queries
while HiveServer2 does not support it. I dont think you can specify any
configuration in hive-site.xml which can make it connect to HiveServer2.
It becomes a blocking issue in case of Sentry where HiveServer2 does the
translation of authenticated users to "hive" user (which is the only user
that can access directories under hive/warehouse when Sentry is ON). The
HiveContext is able to access the metastore and then tries to access the
files under hive warehouse directory where i fails with permission error:
*org.apache.hadoop.security.AccessControlException: Permission denied:
user=kakn, access=READ_EXECUTE,
inode="/user/hive/warehouse/rt_freewheel_mastering.db/digital_profile_cluster_in":hive:hive:drwxrwx--t*
On Tue, Mar 24, 2015 at 1:43 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> spark-submit --files /path/to/hive-site.xml
>
> On Tue, Mar 24, 2015 at 10:31 AM, Udit Mehta <um...@groupon.com> wrote:
> > Another question related to this, how can we propagate the hive-site.xml
> to
> > all workers when running in the yarn cluster mode?
> >
> > On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin <va...@cloudera.com>
> > wrote:
> >>
> >> It does neither. If you provide a Hive configuration to Spark,
> >> HiveContext will connect to your metastore server, otherwise it will
> >> create its own metastore in the working directory (IIRC).
> >>
> >> On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 <ni...@gmail.com>
> >> wrote:
> >> > I am wondering if HiveContext connects to HiveServer2 or does it work
> >> > though
> >> > Hive CLI. The reason I am asking is because Cloudera has deprecated
> Hive
> >> > CLI.
> >> >
> >> > If the connection is through HiverServer2, is there a way to specify
> >> > user
> >> > credentials?
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html
> >> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> > For additional commands, e-mail: user-help@spark.apache.org
> >> >
> >>
> >>
> >>
> >> --
> >> Marcelo
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >
>
>
>
> --
> Marcelo
>
Re: Does HiveContext connect to HiveServer2?
Posted by Marcelo Vanzin <va...@cloudera.com>.
spark-submit --files /path/to/hive-site.xml
On Tue, Mar 24, 2015 at 10:31 AM, Udit Mehta <um...@groupon.com> wrote:
> Another question related to this, how can we propagate the hive-site.xml to
> all workers when running in the yarn cluster mode?
>
> On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>>
>> It does neither. If you provide a Hive configuration to Spark,
>> HiveContext will connect to your metastore server, otherwise it will
>> create its own metastore in the working directory (IIRC).
>>
>> On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 <ni...@gmail.com>
>> wrote:
>> > I am wondering if HiveContext connects to HiveServer2 or does it work
>> > though
>> > Hive CLI. The reason I am asking is because Cloudera has deprecated Hive
>> > CLI.
>> >
>> > If the connection is through HiverServer2, is there a way to specify
>> > user
>> > credentials?
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Does HiveContext connect to HiveServer2?
Posted by Udit Mehta <um...@groupon.com>.
Another question related to this, how can we propagate the hive-site.xml to
all workers when running in the yarn cluster mode?
On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> It does neither. If you provide a Hive configuration to Spark,
> HiveContext will connect to your metastore server, otherwise it will
> create its own metastore in the working directory (IIRC).
>
> On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 <ni...@gmail.com>
> wrote:
> > I am wondering if HiveContext connects to HiveServer2 or does it work
> though
> > Hive CLI. The reason I am asking is because Cloudera has deprecated Hive
> > CLI.
> >
> > If the connection is through HiverServer2, is there a way to specify user
> > credentials?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Does HiveContext connect to HiveServer2?
Posted by Marcelo Vanzin <va...@cloudera.com>.
It does neither. If you provide a Hive configuration to Spark,
HiveContext will connect to your metastore server, otherwise it will
create its own metastore in the working directory (IIRC).
On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 <ni...@gmail.com> wrote:
> I am wondering if HiveContext connects to HiveServer2 or does it work though
> Hive CLI. The reason I am asking is because Cloudera has deprecated Hive
> CLI.
>
> If the connection is through HiverServer2, is there a way to specify user
> credentials?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Does HiveContext connect to HiveServer2?
Posted by nitinkak001 <ni...@gmail.com>.
Hey, I have exactly this question. Did you get an answer to it?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200p23431.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org