Re: access spark thrift server from another spark session

[Takeshi Yamamuro <li...@gmail.com>]

-dev +user

Hi,

Have you try to share a session by
`spark.sql.hive.thriftServer.singleSession`?

// maropu

On Tue, Oct 4, 2016 at 6:10 AM, Herman Yu <he...@teeupdata.com> wrote:

>
> I built spark data frame/dataset on top of several hive tables, and then
> registered dataframe/dataset as temporary tables, as well as exposed the
> temporary table through spark thrift server. Now the question is, this
> temporary table is only visible to the same spark session. How do I make
> the registered temporary table available to another spark session?
>
> I have tried manually set the hive.server2.thrift.port in the new spark
> session,but the temporary table is still not seen. I also tried connecting
> to thrift server through JDBC from the 2nd spark session, it doesn’t work
> either (seems spark doesn’t like the idea accessing hive through JDBC).
>
> so the questions is: is it possible to share data frame/dataset based
> temporary tables through Spark thrift server between multiple spark
> sessions?
>
> Thanks
> Herman.
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>


-- 
---
Takeshi Yamamuro

Re: access spark thrift server from another spark session

[Herman Yu <he...@teeupdata.com>]

Yes, I did set spark.sql.hive.thriftServer.singleSession to true in spark-defaults.conf of both spark sessions. after starting the 2nd spark session, I manually set hive.server2.thrift.port to the spark thrift port started within the 1st spark session, the temporary table is still not visible.


The temp table is visible when I connect to the spark thrift server using beeline ( expecting that it should also work with other JDBC clients even though I didn’t test), I am just not able to make it available to another spark session, or, is it possible?


Thanks
Herman


> On Oct 3, 2016, at 22:02, ayan guha <gu...@gmail.com> wrote:
> 
> I do not think you can see temp tables from other application, like thrift. You need save the tables in hive and then they will be visible through thrift. Thrift uses hive metastore. However temp tables do not make to central metastore until saved.
> 
> On 4 Oct 2016 11:44, "Takeshi Yamamuro" <linguin.m.s@gmail.com <ma...@gmail.com>> wrote:
> -dev +user
> 
> Hi,
> 
> Have you try to share a session by `spark.sql.hive.thriftServer.singleSession`?
> 
> // maropu
> 
> On Tue, Oct 4, 2016 at 6:10 AM, Herman Yu <herman.yu@teeupdata.com <ma...@teeupdata.com>> wrote:
> 
> I built spark data frame/dataset on top of several hive tables, and then registered dataframe/dataset as temporary tables, as well as exposed the temporary table through spark thrift server. Now the question is, this temporary table is only visible to the same spark session. How do I make the registered temporary table available to another spark session?
> 
> I have tried manually set the hive.server2.thrift.port in the new spark session,but the temporary table is still not seen. I also tried connecting to thrift server through JDBC from the 2nd spark session, it doesn’t work either (seems spark doesn’t like the idea accessing hive through JDBC).
> 
> so the questions is: is it possible to share data frame/dataset based temporary tables through Spark thrift server between multiple spark sessions?
> 
> Thanks
> Herman.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> 
> 
> 
> 
> -- 
> ---
> Takeshi Yamamuro

Re: access spark thrift server from another spark session

[ayan guha <gu...@gmail.com>]

I do not think you can see temp tables from other application, like thrift.
You need save the tables in hive and then they will be visible through
thrift. Thrift uses hive metastore. However temp tables do not make to
central metastore until saved.
On 4 Oct 2016 11:44, "Takeshi Yamamuro" <li...@gmail.com> wrote:

> -dev +user
>
> Hi,
>
> Have you try to share a session by `spark.sql.hive.thriftServer.
> singleSession`?
>
> // maropu
>
> On Tue, Oct 4, 2016 at 6:10 AM, Herman Yu <he...@teeupdata.com> wrote:
>
>>
>> I built spark data frame/dataset on top of several hive tables, and then
>> registered dataframe/dataset as temporary tables, as well as exposed the
>> temporary table through spark thrift server. Now the question is, this
>> temporary table is only visible to the same spark session. How do I make
>> the registered temporary table available to another spark session?
>>
>> I have tried manually set the hive.server2.thrift.port in the new spark
>> session,but the temporary table is still not seen. I also tried connecting
>> to thrift server through JDBC from the 2nd spark session, it doesn’t work
>> either (seems spark doesn’t like the idea accessing hive through JDBC).
>>
>> so the questions is: is it possible to share data frame/dataset based
>> temporary tables through Spark thrift server between multiple spark
>> sessions?
>>
>> Thanks
>> Herman.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> ---
> Takeshi Yamamuro
>