You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Sarnath <st...@gmail.com> on 2015/11/25 19:58:37 UTC
Jdbc server
Hi All,
Good morning!
We are implementing an OLAP engine on Hadoop. We store the aggregations as
documents in elastic search. Elastic search provides a JSON based query DSL
that can be used via REST API to filter and select these aggregations.
We want to provide an SQL interface on top of this...and power the queries
through the cubes that we store in Elastic Search.
I am seeking your guidance on how to go about it using calcite and avatica.
We already developed a model.json for ES and able to run SQL on Elastic
search from sqlline that accompanies calcite. But what we would like to do
is deploy a JDBC server and automatically serve queries either from elastic
search or through hiveserver. We are not clear how to roll out our own jdbc
client and server... Can some1 explain this please?
Also in the calcite presentation, I found references to lattice and tiles.
I am yet to find out what are their roles are within calcite and how our
project could benefit out from these. If one of you could throw some light,
I would be grateful.
Thanks,
Best,
Sarnath
Best,
Sarnath
Re: Jdbc server
Posted by Sarnath <st...@gmail.com>.
We wrote a tech blog on our work.
Interested can check here:
http://www.hcltech.com/blogs/engineering-and-rd-services/olap-cubing-big-data
We may rephrase few things in the blog next week. But this will give you
the general direction if our approach.
Re: Jdbc server
Posted by Sarnath <st...@gmail.com>.
Thanks. I think I have a lot to catch up. Hope will be able to do in 2nd
half of this month....
Best,
Sarnath
On Nov 27, 2015 11:09 PM, "Rajat Venkatesh" <rv...@qubole.com> wrote:
> The optimizer runs in the same JVM as the JDBC code. Very similar to the
> org.apache.calcite.jdbc.Driver which we used as a template.
>
> On Friday 27 November 2015, Sarnath <st...@gmail.com> wrote:
>
>> Hi Rajat,
>> Good to connect with you. Good to know about your work.
>> I did not quite understand when you said you have jdbc access through
>> avatica but no jdbc server and a fat jar....
>> Also, we will publish a blog soon on an experiment we did. I will keep
>> you posted on that.
>> Good luck!
>> Best,
>> Sarnath
>> On Nov 27, 2015 5:51 PM, "Rajat Venkatesh" <rv...@qubole.com> wrote:
>>
>>> Hi Sarnath,
>>> We are building something similar at Qubole. The difference is that our
>>> customers store their cubes and vies in a relational database - usually AWS
>>> Redshift. So their application continues to refer to tables in Hive but a
>>> cube or a view is used to answer the query if possible. The queries are
>>> routed to the right data store.
>>> All this is built on top of the calcite optimizer and we also have JDBC
>>> access using Avatica. We don't have a JDBC server yet but a fat jar.
>>> Seems like you working on something similar. I'll gladly exchange notes.
>>> We are planning to open source it as well. So if you wish to use it as a
>>> platform for your experiments, I can help with that as well.
>>>
>>> On Thursday 26 November 2015, Sarnath <st...@gmail.com> wrote:
>>>
>>>> Hi Julian,
>>>> Thanks for the details and thanks a lot for your time. It looks pretty
>>>> simple. We will explore more and get back here if we cannot resolve it
>>>> through the docs.
>>>> Again, thanks a lot for your time and your help.
>>>> Best,
>>>> Sarnath
>>>>
>>>
>>>
>>> --
>>> Rajat Venkatesh | Engg Lead
>>> Qubole Inc | www.qubole.com
>>>
>>>
>
> --
> Rajat Venkatesh | Engg Lead
> Qubole Inc | www.qubole.com
>
>
Re: Jdbc server
Posted by Rajat Venkatesh <rv...@qubole.com>.
The optimizer runs in the same JVM as the JDBC code. Very similar to the
org.apache.calcite.jdbc.Driver which we used as a template.
On Friday 27 November 2015, Sarnath <st...@gmail.com> wrote:
> Hi Rajat,
> Good to connect with you. Good to know about your work.
> I did not quite understand when you said you have jdbc access through
> avatica but no jdbc server and a fat jar....
> Also, we will publish a blog soon on an experiment we did. I will keep you
> posted on that.
> Good luck!
> Best,
> Sarnath
> On Nov 27, 2015 5:51 PM, "Rajat Venkatesh" <rvenkatesh@qubole.com
> <javascript:_e(%7B%7D,'cvml','rvenkatesh@qubole.com');>> wrote:
>
>> Hi Sarnath,
>> We are building something similar at Qubole. The difference is that our
>> customers store their cubes and vies in a relational database - usually AWS
>> Redshift. So their application continues to refer to tables in Hive but a
>> cube or a view is used to answer the query if possible. The queries are
>> routed to the right data store.
>> All this is built on top of the calcite optimizer and we also have JDBC
>> access using Avatica. We don't have a JDBC server yet but a fat jar.
>> Seems like you working on something similar. I'll gladly exchange notes.
>> We are planning to open source it as well. So if you wish to use it as a
>> platform for your experiments, I can help with that as well.
>>
>> On Thursday 26 November 2015, Sarnath <stellium@gmail.com
>> <javascript:_e(%7B%7D,'cvml','stellium@gmail.com');>> wrote:
>>
>>> Hi Julian,
>>> Thanks for the details and thanks a lot for your time. It looks pretty
>>> simple. We will explore more and get back here if we cannot resolve it
>>> through the docs.
>>> Again, thanks a lot for your time and your help.
>>> Best,
>>> Sarnath
>>>
>>
>>
>> --
>> Rajat Venkatesh | Engg Lead
>> Qubole Inc | www.qubole.com
>>
>>
--
Rajat Venkatesh | Engg Lead
Qubole Inc | www.qubole.com
Re: Jdbc server
Posted by Sarnath <st...@gmail.com>.
Hi Rajat,
Good to connect with you. Good to know about your work.
I did not quite understand when you said you have jdbc access through
avatica but no jdbc server and a fat jar....
Also, we will publish a blog soon on an experiment we did. I will keep you
posted on that.
Good luck!
Best,
Sarnath
On Nov 27, 2015 5:51 PM, "Rajat Venkatesh" <rv...@qubole.com> wrote:
> Hi Sarnath,
> We are building something similar at Qubole. The difference is that our
> customers store their cubes and vies in a relational database - usually AWS
> Redshift. So their application continues to refer to tables in Hive but a
> cube or a view is used to answer the query if possible. The queries are
> routed to the right data store.
> All this is built on top of the calcite optimizer and we also have JDBC
> access using Avatica. We don't have a JDBC server yet but a fat jar.
> Seems like you working on something similar. I'll gladly exchange notes.
> We are planning to open source it as well. So if you wish to use it as a
> platform for your experiments, I can help with that as well.
>
> On Thursday 26 November 2015, Sarnath <st...@gmail.com> wrote:
>
>> Hi Julian,
>> Thanks for the details and thanks a lot for your time. It looks pretty
>> simple. We will explore more and get back here if we cannot resolve it
>> through the docs.
>> Again, thanks a lot for your time and your help.
>> Best,
>> Sarnath
>>
>
>
> --
> Rajat Venkatesh | Engg Lead
> Qubole Inc | www.qubole.com
>
>
Re: Jdbc server
Posted by Dhruv Gohil <yo...@gmail.com>.
+1 , We are in need to find a reference code where we can take decision
transparently just where "where conditions" which backend-data-store the
results should be served.
On Friday 27 November 2015 05:51 PM, Rajat Venkatesh wrote:
> Hi Sarnath,
> We are building something similar at Qubole. The difference is that our
> customers store their cubes and vies in a relational database - usually AWS
> Redshift. So their application continues to refer to tables in Hive but a
> cube or a view is used to answer the query if possible. The queries are
> routed to the right data store.
> All this is built on top of the calcite optimizer and we also have JDBC
> access using Avatica. We don't have a JDBC server yet but a fat jar.
> Seems like you working on something similar. I'll gladly exchange notes. We
> are planning to open source it as well. So if you wish to use it as a
> platform for your experiments, I can help with that as well.
>
> On Thursday 26 November 2015, Sarnath <st...@gmail.com> wrote:
>
>> Hi Julian,
>> Thanks for the details and thanks a lot for your time. It looks pretty
>> simple. We will explore more and get back here if we cannot resolve it
>> through the docs.
>> Again, thanks a lot for your time and your help.
>> Best,
>> Sarnath
>>
>
Re: Jdbc server
Posted by Rajat Venkatesh <rv...@qubole.com>.
Hi Sarnath,
We are building something similar at Qubole. The difference is that our
customers store their cubes and vies in a relational database - usually AWS
Redshift. So their application continues to refer to tables in Hive but a
cube or a view is used to answer the query if possible. The queries are
routed to the right data store.
All this is built on top of the calcite optimizer and we also have JDBC
access using Avatica. We don't have a JDBC server yet but a fat jar.
Seems like you working on something similar. I'll gladly exchange notes. We
are planning to open source it as well. So if you wish to use it as a
platform for your experiments, I can help with that as well.
On Thursday 26 November 2015, Sarnath <st...@gmail.com> wrote:
> Hi Julian,
> Thanks for the details and thanks a lot for your time. It looks pretty
> simple. We will explore more and get back here if we cannot resolve it
> through the docs.
> Again, thanks a lot for your time and your help.
> Best,
> Sarnath
>
--
Rajat Venkatesh | Engg Lead
Qubole Inc | www.qubole.com
Re: Jdbc server
Posted by Sarnath <st...@gmail.com>.
Hi Julian,
Thanks for the details and thanks a lot for your time. It looks pretty
simple. We will explore more and get back here if we cannot resolve it
through the docs.
Again, thanks a lot for your time and your help.
Best,
Sarnath
Re: Jdbc server
Posted by Julian Hyde <jh...@apache.org>.
First, see the documentation at http://calcite.apache.org/docs/avatica_overview.html#http-server.
The simplest way to start the server is to run
$ java <classpath> org.apache.calcite.avatica.server.Main <args>
as a command line. It creates an implementation of Meta, then wraps it in a Jetty handler, then invokes Jetty. There are arguments that can change the port, the implementation of Meta, and so forth. See Main.java for the details.
(You can use any implementation of Meta. If your database is based on Calcite, you will probably use org.apache.calcite.jdbc.CalciteMetaImpl. But Avatica does not require that you use Calcite.)
Once the server is started, you can connect using the remote client driver. Make sure that org.apache.calcite.avatica.remote.Driver is on your path. Then connect to “jdbc:avatica:remote://hostname” using JDBC.
(Others, please chime in if you see errors in the above instructions.)
Julian
> On Nov 25, 2015, at 10:58 AM, Sarnath <st...@gmail.com> wrote:
>
> Hi All,
>
> Good morning!
>
> We are implementing an OLAP engine on Hadoop. We store the aggregations as
> documents in elastic search. Elastic search provides a JSON based query DSL
> that can be used via REST API to filter and select these aggregations.
>
> We want to provide an SQL interface on top of this...and power the queries
> through the cubes that we store in Elastic Search.
>
> I am seeking your guidance on how to go about it using calcite and avatica.
> We already developed a model.json for ES and able to run SQL on Elastic
> search from sqlline that accompanies calcite. But what we would like to do
> is deploy a JDBC server and automatically serve queries either from elastic
> search or through hiveserver. We are not clear how to roll out our own jdbc
> client and server... Can some1 explain this please?
>
> Also in the calcite presentation, I found references to lattice and tiles.
> I am yet to find out what are their roles are within calcite and how our
> project could benefit out from these. If one of you could throw some light,
> I would be grateful.
>
> Thanks,
> Best,
> Sarnath
>
> Best,
> Sarnath