You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2017/08/10 20:13:51 UTC
Does Spark SQL uses Calcite?
Hi All,
Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has
catalyst which would generate its own logical plans, physical plans and
other optimizations.
Thanks,
Kant
Re: Does Spark SQL uses Calcite?
Posted by kant kodali <ka...@gmail.com>.
Hi Jules,
I am looking to connect to Spark via JDBC so I can run Spark SQL queries
via JDBC but not use SPARK SQL to connect to other JDBC sources.
Thanks!
On Sat, Aug 19, 2017 at 5:54 PM, Jules Damji <dm...@comcast.net> wrote:
> Try this link to see how you may connect https://docs.databricks.com/
> spark/latest/data-sources/sql-databases.html
>
> Cheers
> Jules
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Aug 19, 2017, at 5:27 PM, kant kodali <ka...@gmail.com> wrote:
>
> Hi Russell,
>
> I went through this https://jaceklaskowski.gitbooks.io/mastering-apache-
> spark/spark-sql-thrift-server.html and I am still a bit confused on what
> hive is doing in here ? Is there any example I can look at on how to talk
> to Spark using Spark SQL JDBC driver alone and not hive ?
>
> Thanks,
> kant
>
>
>
>
>
> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer <
> russell.spitzer@gmail.com> wrote:
>
>> You don't have to go through hive. It's just spark sql. The application
>> is just a forked hive thrift server.
>>
>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali <ka...@gmail.com> wrote:
>>
>>> @Ryan it looks like if I enable thrift server I need to go through hive.
>>> I was talking more about having JDBC connector for Spark SQL itself other
>>> words not going through hive.
>>>
>>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali <ka...@gmail.com> wrote:
>>>
>>>> @Ryan Does it work with Spark SQL 2.1.1?
>>>>
>>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
>>>>
>>>>> the thrift server is a jdbc server, Kanth
>>>>>
>>>>> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>>>>>
>>>>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>>>>>
>>>>>> Yes, it's more used in Hive than Spark
>>>>>>
>>>>>> Sent from my iPhone
>>>>>> Pardon the dumb thumb typos :)
>>>>>>
>>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>>>>>> vsathishkumaran@gmail.com> wrote:
>>>>>>
>>>>>> I think it is for hive dependency.
>>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>>>>>> being used?
>>>>>>>
>>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>>>>>> vsathishkumaran@gmail.com> wrote:
>>>>>>>
>>>>>>>> Spark SQL doesn't use Calcite
>>>>>>>>
>>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark
>>>>>>>>> SQL has catalyst which would generate its own logical plans, physical plans
>>>>>>>>> and other optimizations.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kant
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>
Re: Does Spark SQL uses Calcite?
Posted by Jules Damji <dm...@comcast.net>.
Try this link to see how you may connect https://docs.databricks.com/spark/latest/data-sources/sql-databases.html
Cheers
Jules
Sent from my iPhone
Pardon the dumb thumb typos :)
> On Aug 19, 2017, at 5:27 PM, kant kodali <ka...@gmail.com> wrote:
>
> Hi Russell,
>
> I went through this https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html and I am still a bit confused on what hive is doing in here ? Is there any example I can look at on how to talk to Spark using Spark SQL JDBC driver alone and not hive ?
>
> Thanks,
> kant
>
>
>
>
>
>> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer <ru...@gmail.com> wrote:
>> You don't have to go through hive. It's just spark sql. The application is just a forked hive thrift server.
>>
>>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali <ka...@gmail.com> wrote:
>>> @Ryan it looks like if I enable thrift server I need to go through hive. I was talking more about having JDBC connector for Spark SQL itself other words not going through hive.
>>>
>>>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali <ka...@gmail.com> wrote:
>>>> @Ryan Does it work with Spark SQL 2.1.1?
>>>>
>>>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
>>>>> the thrift server is a jdbc server, Kanth
>>>>>
>>>>>> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>>>>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>>>>>>
>>>>>>> Yes, it's more used in Hive than Spark
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>> Pardon the dumb thumb typos :)
>>>>>>>
>>>>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I think it is for hive dependency.
>>>>>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>>>>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is being used?
>>>>>>>>>
>>>>>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>>>>>>>>>> Spark SQL doesn't use Calcite
>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Kant
>>>>>>>>>
>>>>>
>>>>
>>>
>
Re: Does Spark SQL uses Calcite?
Posted by kant kodali <ka...@gmail.com>.
Hi Russell,
I went through this
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html
and I am still a bit confused on what hive is doing in here ? Is there any
example I can look at on how to talk to Spark using Spark SQL JDBC driver
alone and not hive ?
Thanks,
kant
On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer <ru...@gmail.com>
wrote:
> You don't have to go through hive. It's just spark sql. The application is
> just a forked hive thrift server.
>
> On Fri, Aug 11, 2017 at 8:53 PM kant kodali <ka...@gmail.com> wrote:
>
>> @Ryan it looks like if I enable thrift server I need to go through hive.
>> I was talking more about having JDBC connector for Spark SQL itself other
>> words not going through hive.
>>
>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali <ka...@gmail.com> wrote:
>>
>>> @Ryan Does it work with Spark SQL 2.1.1?
>>>
>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
>>>
>>>> the thrift server is a jdbc server, Kanth
>>>>
>>>> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>>>>
>>>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>>>>
>>>>> Yes, it's more used in Hive than Spark
>>>>>
>>>>> Sent from my iPhone
>>>>> Pardon the dumb thumb typos :)
>>>>>
>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>>>>> vsathishkumaran@gmail.com> wrote:
>>>>>
>>>>> I think it is for hive dependency.
>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>>>>> being used?
>>>>>>
>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>>>>> vsathishkumaran@gmail.com> wrote:
>>>>>>
>>>>>>> Spark SQL doesn't use Calcite
>>>>>>>
>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark
>>>>>>>> SQL has catalyst which would generate its own logical plans, physical plans
>>>>>>>> and other optimizations.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kant
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
Re: Does Spark SQL uses Calcite?
Posted by Russell Spitzer <ru...@gmail.com>.
You don't have to go through hive. It's just spark sql. The application is
just a forked hive thrift server.
On Fri, Aug 11, 2017 at 8:53 PM kant kodali <ka...@gmail.com> wrote:
> @Ryan it looks like if I enable thrift server I need to go through hive. I
> was talking more about having JDBC connector for Spark SQL itself other
> words not going through hive.
>
> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali <ka...@gmail.com> wrote:
>
>> @Ryan Does it work with Spark SQL 2.1.1?
>>
>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
>>
>>> the thrift server is a jdbc server, Kanth
>>>
>>> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>>>
>>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>>>
>>>> Yes, it's more used in Hive than Spark
>>>>
>>>> Sent from my iPhone
>>>> Pardon the dumb thumb typos :)
>>>>
>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>>>> vsathishkumaran@gmail.com> wrote:
>>>>
>>>> I think it is for hive dependency.
>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>>>>
>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>>>> being used?
>>>>>
>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>>>> vsathishkumaran@gmail.com> wrote:
>>>>>
>>>>>> Spark SQL doesn't use Calcite
>>>>>>
>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark
>>>>>>> SQL has catalyst which would generate its own logical plans, physical plans
>>>>>>> and other optimizations.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kant
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Does Spark SQL uses Calcite?
Posted by kant kodali <ka...@gmail.com>.
@Ryan it looks like if I enable thrift server I need to go through hive. I
was talking more about having JDBC connector for Spark SQL itself other
words not going through hive.
On Fri, Aug 11, 2017 at 6:50 PM, kant kodali <ka...@gmail.com> wrote:
> @Ryan Does it work with Spark SQL 2.1.1?
>
> On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
>
>> the thrift server is a jdbc server, Kanth
>>
>> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>>
>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>>
>>> Yes, it's more used in Hive than Spark
>>>
>>> Sent from my iPhone
>>> Pardon the dumb thumb typos :)
>>>
>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>>> vsathishkumaran@gmail.com> wrote:
>>>
>>> I think it is for hive dependency.
>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>>>
>>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>>> being used?
>>>>
>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>>> vsathishkumaran@gmail.com> wrote:
>>>>
>>>>> Spark SQL doesn't use Calcite
>>>>>
>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>>>>>> has catalyst which would generate its own logical plans, physical plans and
>>>>>> other optimizations.
>>>>>>
>>>>>> Thanks,
>>>>>> Kant
>>>>>>
>>>>>
>>>>
>>
>
Re: Does Spark SQL uses Calcite?
Posted by kant kodali <ka...@gmail.com>.
@Ryan Does it work with Spark SQL 2.1.1?
On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ry...@gmail.com> wrote:
> the thrift server is a jdbc server, Kanth
>
> On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
>
>> I also wonder why there isn't a jdbc connector for spark sql?
>>
>> Sent from my iPhone
>>
>> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>>
>> Yes, it's more used in Hive than Spark
>>
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>> vsathishkumaran@gmail.com> wrote:
>>
>> I think it is for hive dependency.
>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>>
>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>> being used?
>>>
>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>> vsathishkumaran@gmail.com> wrote:
>>>
>>>> Spark SQL doesn't use Calcite
>>>>
>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>>>>> has catalyst which would generate its own logical plans, physical plans and
>>>>> other optimizations.
>>>>>
>>>>> Thanks,
>>>>> Kant
>>>>>
>>>>
>>>
>
Re: Does Spark SQL uses Calcite?
Posted by Ryan <ry...@gmail.com>.
the thrift server is a jdbc server, Kanth
On Fri, Aug 11, 2017 at 2:51 PM, <ka...@gmail.com> wrote:
> I also wonder why there isn't a jdbc connector for spark sql?
>
> Sent from my iPhone
>
> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>
> Yes, it's more used in Hive than Spark
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
> vsathishkumaran@gmail.com> wrote:
>
> I think it is for hive dependency.
> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>
>> Since I see a calcite dependency in Spark I wonder where Calcite is being
>> used?
>>
>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>> vsathishkumaran@gmail.com> wrote:
>>
>>> Spark SQL doesn't use Calcite
>>>
>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>>>> has catalyst which would generate its own logical plans, physical plans and
>>>> other optimizations.
>>>>
>>>> Thanks,
>>>> Kant
>>>>
>>>
>>
Re: Does Spark SQL uses Calcite?
Posted by ka...@gmail.com.
I also wonder why there isn't a jdbc connector for spark sql?
Sent from my iPhone
> On Aug 10, 2017, at 2:45 PM, Jules Damji <dm...@comcast.net> wrote:
>
> Yes, it's more used in Hive than Spark
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>>
>> I think it is for hive dependency.
>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>>> Since I see a calcite dependency in Spark I wonder where Calcite is being used?
>>>
>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>>>> Spark SQL doesn't use Calcite
>>>>
>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>>>> Hi All,
>>>>>
>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations.
>>>>>
>>>>> Thanks,
>>>>> Kant
>>>
Re: Does Spark SQL uses Calcite?
Posted by Jules Damji <dm...@comcast.net>.
Yes, it's more used in Hive than Spark
Sent from my iPhone
Pardon the dumb thumb typos :)
> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>
> I think it is for hive dependency.
>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
>> Since I see a calcite dependency in Spark I wonder where Calcite is being used?
>>
>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
>>> Spark SQL doesn't use Calcite
>>>
>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations.
>>>>
>>>> Thanks,
>>>> Kant
>>
Re: Does Spark SQL uses Calcite?
Posted by Sathish Kumaran Vairavelu <vs...@gmail.com>.
I think it is for hive dependency.
On Thu, Aug 10, 2017 at 4:14 PM kant kodali <ka...@gmail.com> wrote:
> Since I see a calcite dependency in Spark I wonder where Calcite is being
> used?
>
> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
> vsathishkumaran@gmail.com> wrote:
>
>> Spark SQL doesn't use Calcite
>>
>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>>> has catalyst which would generate its own logical plans, physical plans and
>>> other optimizations.
>>>
>>> Thanks,
>>> Kant
>>>
>>
>
Re: Does Spark SQL uses Calcite?
Posted by kant kodali <ka...@gmail.com>.
Since I see a calcite dependency in Spark I wonder where Calcite is being
used?
On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
vsathishkumaran@gmail.com> wrote:
> Spark SQL doesn't use Calcite
>
> On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
>
>> Hi All,
>>
>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has
>> catalyst which would generate its own logical plans, physical plans and
>> other optimizations.
>>
>> Thanks,
>> Kant
>>
>
Re: Does Spark SQL uses Calcite?
Posted by Sathish Kumaran Vairavelu <vs...@gmail.com>.
Spark SQL doesn't use Calcite
On Thu, Aug 10, 2017 at 3:14 PM kant kodali <ka...@gmail.com> wrote:
> Hi All,
>
> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has
> catalyst which would generate its own logical plans, physical plans and
> other optimizations.
>
> Thanks,
> Kant
>