You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Natu Lauchande <nl...@gmail.com> on 2016/04/12 10:48:36 UTC
Can i have a hive context and sql context in the same app ?
Hi,
Is it possible to have both a sqlContext and a hiveContext in the same
application ?
If yes would there be any performance pernalties of doing so.
Regards,
Natu
Re: Can i have a hive context and sql context in the same app ?
Posted by Gourav Sengupta <go...@gmail.com>.
allowing multiple SPARK context is different from hiveContext and
sqlContext. hiveContext has more methods and properties than sqlContext and
all the methods and properties.
In the GUI you will be able to see SQL, SQL1, .... tabs when multiple
sqlContexts are active.
But why would you want that?
Regards,
Gourav
On Tue, Apr 12, 2016 at 12:46 PM, Sun, Rui <ru...@intel.com> wrote:
> val ALLOW_MULTIPLE_CONTEXTS =
> booleanConf("spark.sql.allowMultipleContexts",
>
> defaultValue = Some(true),
>
> doc = "When set to true, creating multiple SQLContexts/HiveContexts is
> allowed." +
>
> "When set to false, only one SQLContext/HiveContext is allowed to be
> created " +
>
> "through the constructor (new SQLContexts/HiveContexts created
> through newSession " +
>
> "method is allowed). Please note that this conf needs to be set in
> Spark Conf. Once" +
>
> "a SQLContext/HiveContext has been created, changing the value of
> this conf will not" +
>
> "have effect.",
>
> isPublic = true)
>
>
>
> I don’t think there is any performance pernalties of doing so.
>
> *From:* Natu Lauchande [mailto:nlauchande@gmail.com]
> *Sent:* Tuesday, April 12, 2016 4:49 PM
> *To:* user@spark.apache.org
> *Subject:* Can i have a hive context and sql context in the same app ?
>
>
>
> Hi,
>
> Is it possible to have both a sqlContext and a hiveContext in the same
> application ?
>
> If yes would there be any performance pernalties of doing so.
>
>
>
> Regards,
>
> Natu
>
RE: Can i have a hive context and sql context in the same app ?
Posted by "Sun, Rui" <ru...@intel.com>.
val ALLOW_MULTIPLE_CONTEXTS = booleanConf("spark.sql.allowMultipleContexts",
defaultValue = Some(true),
doc = "When set to true, creating multiple SQLContexts/HiveContexts is allowed." +
"When set to false, only one SQLContext/HiveContext is allowed to be created " +
"through the constructor (new SQLContexts/HiveContexts created through newSession " +
"method is allowed). Please note that this conf needs to be set in Spark Conf. Once" +
"a SQLContext/HiveContext has been created, changing the value of this conf will not" +
"have effect.",
isPublic = true)
I don’t think there is any performance pernalties of doing so.
From: Natu Lauchande [mailto:nlauchande@gmail.com]
Sent: Tuesday, April 12, 2016 4:49 PM
To: user@spark.apache.org
Subject: Can i have a hive context and sql context in the same app ?
Hi,
Is it possible to have both a sqlContext and a hiveContext in the same application ?
If yes would there be any performance pernalties of doing so.
Regards,
Natu
Re: Can i have a hive context and sql context in the same app ?
Posted by Michael Armbrust <mi...@databricks.com>.
You don't need multiple contexts to do this:
http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases
On Tue, Apr 12, 2016 at 4:05 PM, Michael Segel <mi...@hotmail.com>
wrote:
> Reading from multiple sources within the same application?
>
> How would you connect to Hive for some data and then reach out to lets say
> Oracle or DB2 for some other data that you may want but isn’t available on
> your cluster?
>
>
> On Apr 12, 2016, at 10:52 AM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
> You can, but I'm not sure why you would want to. If you want to isolate
> different users just use hiveContext.newSession().
>
> On Tue, Apr 12, 2016 at 1:48 AM, Natu Lauchande <nl...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Is it possible to have both a sqlContext and a hiveContext in the same
>> application ?
>>
>> If yes would there be any performance pernalties of doing so.
>>
>> Regards,
>> Natu
>>
>
>
>
Re: Can i have a hive context and sql context in the same app ?
Posted by Michael Armbrust <mi...@databricks.com>.
You can, but I'm not sure why you would want to. If you want to isolate
different users just use hiveContext.newSession().
On Tue, Apr 12, 2016 at 1:48 AM, Natu Lauchande <nl...@gmail.com>
wrote:
> Hi,
>
> Is it possible to have both a sqlContext and a hiveContext in the same
> application ?
>
> If yes would there be any performance pernalties of doing so.
>
> Regards,
> Natu
>