You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Charles Givre <cg...@gmail.com> on 2020/01/17 18:29:36 UTC
Calcite Adapter Question
Hello Calcite Devs!
My name is Charles Givre and I'm the PMC Chair for Apache Drill, which uses Calcite for query planning among other things. I'm working on extending the number of systems that Drill can connect to and I saw that Calcite has a number of adapters for various systems like Cassandra and Elasticsearch.
Could anyone point me to some resources as to how these adapters can be used (or extended) so that Drill could use them?
Thank you very much!
-- C
Re: Calcite Adapter Question
Posted by Julian Hyde <jh...@apache.org>.
I know that Drill uses Calcite for query planning, and therefore I expect that Drill’s adapter model is probably fairly similar to Calcite’s. In fact, I have long wished that we used the same adapter model, so that we could share adapters.
Drill already uses Calcite as a runtime library, so that shouldn’t be a problem.
There is a page that documents adapters[1]. The heart of each adapter is a schema factory[2], which provides the metadata necessary to validate and plan a query. You should probably write a meta-adapter that takes a Calcite schema factory and converts it into a Drill adapter.
Each adapter has a test, e.g. CassandraAdapterTest. It is a good idea to copy-paste that test into Drill’s test suite so that you know you are inheriting the same functionality.
Julian
[1] https://calcite.apache.org/docs/adapter.html <https://calcite.apache.org/docs/adapter.html>
[2] https://calcite.apache.org/apidocs/org/apache/calcite/schema/SchemaFactory.html <https://calcite.apache.org/apidocs/org/apache/calcite/schema/SchemaFactory.html>
> On Jan 17, 2020, at 10:52 AM, Andrei Sereda <andrei@sereda.cc <ma...@sereda.cc>> wrote:
>
> Hi Charles,
>
> There is some documentation here
> https://calcite.apache.org/docs/adapter.html <https://calcite.apache.org/docs/adapter.html> it describes how to setup and
> use an adapter.
>
> Note that you can only use those adapters within calcite runtime and not as
> a standalone library.
>
> Typical adapter would expose collection (for Mongo), index (for ES), region
> (for Geode) etc. as a calcite schema / table which you can query using
> relational algebra.
>
> How exactly do you want to extend existing adapters ?
>
> Hope that helps.
>
> Regards,
> Andrei.
>
>
>
>
> On Fri, Jan 17, 2020 at 1:29 PM Charles Givre <cgivre@gmail.com <ma...@gmail.com>> wrote:
>
>> Hello Calcite Devs!
>> My name is Charles Givre and I'm the PMC Chair for Apache Drill, which
>> uses Calcite for query planning among other things. I'm working on
>> extending the number of systems that Drill can connect to and I saw that
>> Calcite has a number of adapters for various systems like Cassandra and
>> Elasticsearch.
>>
>> Could anyone point me to some resources as to how these adapters can be
>> used (or extended) so that Drill could use them?
>> Thank you very much!
>> -- C
Re: Calcite Adapter Question
Posted by Julian Hyde <jh...@apache.org>.
I know that Drill uses Calcite for query planning, and therefore I expect that Drill’s adapter model is probably fairly similar to Calcite’s. In fact, I have long wished that we used the same adapter model, so that we could share adapters.
Drill already uses Calcite as a runtime library, so that shouldn’t be a problem.
There is a page that documents adapters[1]. The heart of each adapter is a schema factory[2], which provides the metadata necessary to validate and plan a query. You should probably write a meta-adapter that takes a Calcite schema factory and converts it into a Drill adapter.
Each adapter has a test, e.g. CassandraAdapterTest. It is a good idea to copy-paste that test into Drill’s test suite so that you know you are inheriting the same functionality.
Julian
[1] https://calcite.apache.org/docs/adapter.html <https://calcite.apache.org/docs/adapter.html>
[2] https://calcite.apache.org/apidocs/org/apache/calcite/schema/SchemaFactory.html <https://calcite.apache.org/apidocs/org/apache/calcite/schema/SchemaFactory.html>
> On Jan 17, 2020, at 10:52 AM, Andrei Sereda <an...@sereda.cc> wrote:
>
> Hi Charles,
>
> There is some documentation here
> https://calcite.apache.org/docs/adapter.html it describes how to setup and
> use an adapter.
>
> Note that you can only use those adapters within calcite runtime and not as
> a standalone library.
>
> Typical adapter would expose collection (for Mongo), index (for ES), region
> (for Geode) etc. as a calcite schema / table which you can query using
> relational algebra.
>
> How exactly do you want to extend existing adapters ?
>
> Hope that helps.
>
> Regards,
> Andrei.
>
>
>
>
> On Fri, Jan 17, 2020 at 1:29 PM Charles Givre <cg...@gmail.com> wrote:
>
>> Hello Calcite Devs!
>> My name is Charles Givre and I'm the PMC Chair for Apache Drill, which
>> uses Calcite for query planning among other things. I'm working on
>> extending the number of systems that Drill can connect to and I saw that
>> Calcite has a number of adapters for various systems like Cassandra and
>> Elasticsearch.
>>
>> Could anyone point me to some resources as to how these adapters can be
>> used (or extended) so that Drill could use them?
>> Thank you very much!
>> -- C
Re: Calcite Adapter Question
Posted by Andrei Sereda <an...@sereda.cc>.
Hi Charles,
There is some documentation here
https://calcite.apache.org/docs/adapter.html it describes how to setup and
use an adapter.
Note that you can only use those adapters within calcite runtime and not as
a standalone library.
Typical adapter would expose collection (for Mongo), index (for ES), region
(for Geode) etc. as a calcite schema / table which you can query using
relational algebra.
How exactly do you want to extend existing adapters ?
Hope that helps.
Regards,
Andrei.
On Fri, Jan 17, 2020 at 1:29 PM Charles Givre <cg...@gmail.com> wrote:
> Hello Calcite Devs!
> My name is Charles Givre and I'm the PMC Chair for Apache Drill, which
> uses Calcite for query planning among other things. I'm working on
> extending the number of systems that Drill can connect to and I saw that
> Calcite has a number of adapters for various systems like Cassandra and
> Elasticsearch.
>
> Could anyone point me to some resources as to how these adapters can be
> used (or extended) so that Drill could use them?
> Thank you very much!
> -- C
Re: Calcite Adapter Question
Posted by Michael Mior <mm...@apache.org>.
A common approach to using the adapters is via a JDBC connection
string which specifies the required parameters for the adapter
(example below in Scala since I had the code handy).
val connString =
"jdbc:calcite:schemaFactory=org.apache.calcite.adapter.cassandra.CassandraSchemaFactory;
schema.host=" +
host + "; schema.port=" + port + ";
schema.keyspace=" + keyspaceName
val connectionProps = new Properties()
connectionProps.put("user", "admin")
connectionProps.put("password", "admin")
val conn = DriverManager.getConnection(connString, connectionProps)
However, I'm guessing this isn't quite what you want. Alternatively,
you can use the adapter's schema factory (e.g. CassandraSchemaFactory)
to construct a schema instance (CassandraSchema). From there you could
use Calcite's RelBuilder (or parse from SQL) to build a query. The
rules for pushdown should automatically be registered in Calcite's
planner if you're using Calcite. I'm not too familiar with Drill's
query planning, but I'm sure something similar would work.
--
Michael Mior
mmior@apache.org
Le ven. 17 janv. 2020 à 13:29, Charles Givre <cg...@gmail.com> a écrit :
>
> Hello Calcite Devs!
> My name is Charles Givre and I'm the PMC Chair for Apache Drill, which uses Calcite for query planning among other things. I'm working on extending the number of systems that Drill can connect to and I saw that Calcite has a number of adapters for various systems like Cassandra and Elasticsearch.
>
> Could anyone point me to some resources as to how these adapters can be used (or extended) so that Drill could use them?
> Thank you very much!
> -- C