Spark thrift server on yarn

[Udit Mehta <um...@groupon.com>]

Hi,

I am trying to start a spark thrift server using the following command on
Spark 1.3.1 running on yarn:



* ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032
--executor-memory 512m --hiveconf
hive.server2.thrift.bind.host=test-host.sn1 --hiveconf
hive.server2.thrift.port=10001 --queue public*
It starts up fine and is able to connect to the hive metastore.
I now need to view some temporary tables using this thrift server so I
start up SparkSql and register a temp table.
But the problem is that I am unable to view the temp table using the
beeline client. I am pretty sure I am going wrong somewhere and the spark
documentation does not clearly say how to run the thrift server in yarn
mode or maybe I missed something.
Could someone tell me how this is to be done or point me to some
documentation?

Thanks in advance,
Udit

Re: Spark thrift server on yarn

[Udit Mehta <um...@groupon.com>]

I registered it in a new Spark SQL CLI. Yeah I thought so too about how the
temp tables were accessible across different applications without using a
job-server. I see that running*
HiveThriftServer2.startWithContext(hiveContext) *within the spark app
starts up a thrift server.

On Tue, Aug 25, 2015 at 5:32 PM, Cheng, Hao <ha...@intel.com> wrote:

> Did you register temp table via the beeline or in a new Spark SQL CLI?
>
>
>
> As I know, the temp table cannot cross the HiveContext.
>
>
>
> Hao
>
>
>
> *From:* Udit Mehta [mailto:umehta@groupon.com]
> *Sent:* Wednesday, August 26, 2015 8:19 AM
> *To:* user
> *Subject:* Spark thrift server on yarn
>
>
>
> Hi,
>
> I am trying to start a spark thrift server using the following command on
> Spark 1.3.1 running on yarn:
>
> * ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032
> --executor-memory 512m --hiveconf
> hive.server2.thrift.bind.host=test-host.sn1 --hiveconf
> hive.server2.thrift.port=10001 --queue public*
>
> It starts up fine and is able to connect to the hive metastore.
>
> I now need to view some temporary tables using this thrift server so I
> start up SparkSql and register a temp table.
>
> But the problem is that I am unable to view the temp table using the
> beeline client. I am pretty sure I am going wrong somewhere and the spark
> documentation does not clearly say how to run the thrift server in yarn
> mode or maybe I missed something.
> Could someone tell me how this is to be done or point me to some
> documentation?
>
> Thanks in advance,
>
> Udit
>

RE: Spark thrift server on yarn

["Cheng, Hao" <ha...@intel.com>]

Did you register temp table via the beeline or in a new Spark SQL CLI?

As I know, the temp table cannot cross the HiveContext.

Hao

From: Udit Mehta [mailto:umehta@groupon.com]
Sent: Wednesday, August 26, 2015 8:19 AM
To: user
Subject: Spark thrift server on yarn

Hi,
I am trying to start a spark thrift server using the following command on Spark 1.3.1 running on yarn:

 ./sbin/start-thriftserver.sh --master yarn://resourcemanager.snc1:8032 --executor-memory 512m --hiveconf hive.server2.thrift.bind.host=test-host.sn1 --hiveconf hive.server2.thrift.port=10001 --queue public
It starts up fine and is able to connect to the hive metastore.
I now need to view some temporary tables using this thrift server so I start up SparkSql and register a temp table.
But the problem is that I am unable to view the temp table using the beeline client. I am pretty sure I am going wrong somewhere and the spark documentation does not clearly say how to run the thrift server in yarn mode or maybe I missed something.
Could someone tell me how this is to be done or point me to some documentation?
Thanks in advance,
Udit