You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Chevalier Julien <jc...@numerigraphe.com> on 2015/03/06 14:43:50 UTC
JDBC data source
Dear List
I Wonder if Drill can read data from a RDBMS through a good old JDBC driver
Examples are welcome.
Best regards
Juju
Re: JDBC data source
Posted by Jacques Nadeau <ja...@apache.org>.
Agreed. As soon as we get back to calcite master I hope someone can pick
this up.
On Mar 6, 2015 10:28 AM, "Julian Hyde" <ju...@gmail.com> wrote:
> Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter.
> Calcite has a JDBC adapter that tries to push down as much of the algebra
> tree to a foreign database (this depends upon what the database is capable
> of), then generates database-specific SQL for the tree that it had managed
> to push.
>
> Drill’s requirements are exactly the same so it seems crazy to not start
> with Calcite’s adapter.
>
> Julian
>
> On Mar 6, 2015, at 10:05 AM, Norris Lee <no...@simba.com> wrote:
>
> > Close but not quite. Alex and I are actually working on an ODBC driver.
> An open source JDBC driver already exists and is in active development.
> >
> > As for a JDBC storage plugin to source data from other JDBC drivers (eg.
> Of RDBMSes), it's been requested several times recently but I don't think
> anyone is working on it. If I recall correctly, some people expressed
> interest in working on it.
> >
> > Norris
> >
> > -----Original Message-----
> > From: Yash Sharma [mailto:yash360@gmail.com]
> > Sent: Friday, March 06, 2015 6:51 AM
> > To: user@drill.apache.org
> > Subject: Re: JDBC data source
> >
> > Drill currently doesn't have a jdbc storage plugin. However it has come
> up in discussions couple of times. As discussed in last hangout Norris and
> Alexander are working on it and we can expect a patch soon.
> >
> > -Via mobile. Excuse brevity.
> > On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
> >
> >> Yash, what you pointed out is a way to produce Drill output through a
> JDBC.
> >> I want Drill to consume data from JDBC.
> >> As Carol clearly explains what I need is a JDBC Storage Plugin.
> >>
> >> Precisely are they any JDBC Storage plugin available ?
> >>
> >> Best regards.
> >>
> >> Juju
> >>
> >>> Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
> >>>
> >>> Hi Juju, check if this can help:
> >>>
> >>>
> >> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
> >> +Apache+Drill+from+SQuirreL
> >>>
> >>> Regards
> >>>
> >>>
> >>>
> >>> On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
> >>> <jc...@numerigraphe.com>
> >>> wrote:
> >>>
> >>>> Dear List
> >>>>
> >>>> I Wonder if Drill can read data from a RDBMS through a good old
> >>>> JDBC
> >> driver
> >>>>
> >>>> Examples are welcome.
> >>>>
> >>>> Best regards
> >>>>
> >>>> Juju
> >>>>
> >>>>
> >>
> >>
>
>
Re: JDBC data source
Posted by Jacques Nadeau <ja...@apache.org>.
Agreed. As soon as we get back to calcite master I hope someone can pick
this up.
On Mar 6, 2015 10:28 AM, "Julian Hyde" <ju...@gmail.com> wrote:
> Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter.
> Calcite has a JDBC adapter that tries to push down as much of the algebra
> tree to a foreign database (this depends upon what the database is capable
> of), then generates database-specific SQL for the tree that it had managed
> to push.
>
> Drill’s requirements are exactly the same so it seems crazy to not start
> with Calcite’s adapter.
>
> Julian
>
> On Mar 6, 2015, at 10:05 AM, Norris Lee <no...@simba.com> wrote:
>
> > Close but not quite. Alex and I are actually working on an ODBC driver.
> An open source JDBC driver already exists and is in active development.
> >
> > As for a JDBC storage plugin to source data from other JDBC drivers (eg.
> Of RDBMSes), it's been requested several times recently but I don't think
> anyone is working on it. If I recall correctly, some people expressed
> interest in working on it.
> >
> > Norris
> >
> > -----Original Message-----
> > From: Yash Sharma [mailto:yash360@gmail.com]
> > Sent: Friday, March 06, 2015 6:51 AM
> > To: user@drill.apache.org
> > Subject: Re: JDBC data source
> >
> > Drill currently doesn't have a jdbc storage plugin. However it has come
> up in discussions couple of times. As discussed in last hangout Norris and
> Alexander are working on it and we can expect a patch soon.
> >
> > -Via mobile. Excuse brevity.
> > On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
> >
> >> Yash, what you pointed out is a way to produce Drill output through a
> JDBC.
> >> I want Drill to consume data from JDBC.
> >> As Carol clearly explains what I need is a JDBC Storage Plugin.
> >>
> >> Precisely are they any JDBC Storage plugin available ?
> >>
> >> Best regards.
> >>
> >> Juju
> >>
> >>> Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
> >>>
> >>> Hi Juju, check if this can help:
> >>>
> >>>
> >> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
> >> +Apache+Drill+from+SQuirreL
> >>>
> >>> Regards
> >>>
> >>>
> >>>
> >>> On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
> >>> <jc...@numerigraphe.com>
> >>> wrote:
> >>>
> >>>> Dear List
> >>>>
> >>>> I Wonder if Drill can read data from a RDBMS through a good old
> >>>> JDBC
> >> driver
> >>>>
> >>>> Examples are welcome.
> >>>>
> >>>> Best regards
> >>>>
> >>>> Juju
> >>>>
> >>>>
> >>
> >>
>
>
Re: JDBC data source
Posted by Julian Hyde <ju...@gmail.com>.
Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter. Calcite has a JDBC adapter that tries to push down as much of the algebra tree to a foreign database (this depends upon what the database is capable of), then generates database-specific SQL for the tree that it had managed to push.
Drill’s requirements are exactly the same so it seems crazy to not start with Calcite’s adapter.
Julian
On Mar 6, 2015, at 10:05 AM, Norris Lee <no...@simba.com> wrote:
> Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development.
>
> As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it.
>
> Norris
>
> -----Original Message-----
> From: Yash Sharma [mailto:yash360@gmail.com]
> Sent: Friday, March 06, 2015 6:51 AM
> To: user@drill.apache.org
> Subject: Re: JDBC data source
>
> Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon.
>
> -Via mobile. Excuse brevity.
> On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
>
>> Yash, what you pointed out is a way to produce Drill output through a JDBC.
>> I want Drill to consume data from JDBC.
>> As Carol clearly explains what I need is a JDBC Storage Plugin.
>>
>> Precisely are they any JDBC Storage plugin available ?
>>
>> Best regards.
>>
>> Juju
>>
>>> Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
>>>
>>> Hi Juju, check if this can help:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
>> +Apache+Drill+from+SQuirreL
>>>
>>> Regards
>>>
>>>
>>>
>>> On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
>>> <jc...@numerigraphe.com>
>>> wrote:
>>>
>>>> Dear List
>>>>
>>>> I Wonder if Drill can read data from a RDBMS through a good old
>>>> JDBC
>> driver
>>>>
>>>> Examples are welcome.
>>>>
>>>> Best regards
>>>>
>>>> Juju
>>>>
>>>>
>>
>>
Re: JDBC data source
Posted by Julian Hyde <ju...@gmail.com>.
Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter. Calcite has a JDBC adapter that tries to push down as much of the algebra tree to a foreign database (this depends upon what the database is capable of), then generates database-specific SQL for the tree that it had managed to push.
Drill’s requirements are exactly the same so it seems crazy to not start with Calcite’s adapter.
Julian
On Mar 6, 2015, at 10:05 AM, Norris Lee <no...@simba.com> wrote:
> Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development.
>
> As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it.
>
> Norris
>
> -----Original Message-----
> From: Yash Sharma [mailto:yash360@gmail.com]
> Sent: Friday, March 06, 2015 6:51 AM
> To: user@drill.apache.org
> Subject: Re: JDBC data source
>
> Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon.
>
> -Via mobile. Excuse brevity.
> On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
>
>> Yash, what you pointed out is a way to produce Drill output through a JDBC.
>> I want Drill to consume data from JDBC.
>> As Carol clearly explains what I need is a JDBC Storage Plugin.
>>
>> Precisely are they any JDBC Storage plugin available ?
>>
>> Best regards.
>>
>> Juju
>>
>>> Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
>>>
>>> Hi Juju, check if this can help:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
>> +Apache+Drill+from+SQuirreL
>>>
>>> Regards
>>>
>>>
>>>
>>> On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
>>> <jc...@numerigraphe.com>
>>> wrote:
>>>
>>>> Dear List
>>>>
>>>> I Wonder if Drill can read data from a RDBMS through a good old
>>>> JDBC
>> driver
>>>>
>>>> Examples are welcome.
>>>>
>>>> Best regards
>>>>
>>>> Juju
>>>>
>>>>
>>
>>
RE: JDBC data source
Posted by Norris Lee <no...@simba.com>.
Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development.
As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it.
Norris
-----Original Message-----
From: Yash Sharma [mailto:yash360@gmail.com]
Sent: Friday, March 06, 2015 6:51 AM
To: user@drill.apache.org
Subject: Re: JDBC data source
Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon.
-Via mobile. Excuse brevity.
On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
> Yash, what you pointed out is a way to produce Drill output through a JDBC.
> I want Drill to consume data from JDBC.
> As Carol clearly explains what I need is a JDBC Storage Plugin.
>
> Precisely are they any JDBC Storage plugin available ?
>
> Best regards.
>
> Juju
>
> > Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
> >
> > Hi Juju, check if this can help:
> >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
> +Apache+Drill+from+SQuirreL
> >
> > Regards
> >
> >
> >
> > On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
> > <jc...@numerigraphe.com>
> > wrote:
> >
> >> Dear List
> >>
> >> I Wonder if Drill can read data from a RDBMS through a good old
> >> JDBC
> driver
> >>
> >> Examples are welcome.
> >>
> >> Best regards
> >>
> >> Juju
> >>
> >>
>
>
RE: JDBC data source
Posted by Norris Lee <no...@simba.com>.
Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development.
As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it.
Norris
-----Original Message-----
From: Yash Sharma [mailto:yash360@gmail.com]
Sent: Friday, March 06, 2015 6:51 AM
To: user@drill.apache.org
Subject: Re: JDBC data source
Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon.
-Via mobile. Excuse brevity.
On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
> Yash, what you pointed out is a way to produce Drill output through a JDBC.
> I want Drill to consume data from JDBC.
> As Carol clearly explains what I need is a JDBC Storage Plugin.
>
> Precisely are they any JDBC Storage plugin available ?
>
> Best regards.
>
> Juju
>
> > Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
> >
> > Hi Juju, check if this can help:
> >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access
> +Apache+Drill+from+SQuirreL
> >
> > Regards
> >
> >
> >
> > On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien
> > <jc...@numerigraphe.com>
> > wrote:
> >
> >> Dear List
> >>
> >> I Wonder if Drill can read data from a RDBMS through a good old
> >> JDBC
> driver
> >>
> >> Examples are welcome.
> >>
> >> Best regards
> >>
> >> Juju
> >>
> >>
>
>
Re: JDBC data source
Posted by Yash Sharma <ya...@gmail.com>.
Drill currently doesn't have a jdbc storage plugin. However it has come up
in discussions couple of times. As discussed in last hangout Norris and
Alexander are working on it and we can expect a patch soon.
-Via mobile. Excuse brevity.
On 6 Mar 2015 8:06 pm, "Chevalier Julien" <jc...@numerigraphe.com> wrote:
> Yash, what you pointed out is a way to produce Drill output through a JDBC.
> I want Drill to consume data from JDBC.
> As Carol clearly explains what I need is a JDBC Storage Plugin.
>
> Precisely are they any JDBC Storage plugin available ?
>
> Best regards.
>
> Juju
>
> > Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
> >
> > Hi Juju, check if this can help:
> >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL
> >
> > Regards
> >
> >
> >
> > On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien <jc...@numerigraphe.com>
> > wrote:
> >
> >> Dear List
> >>
> >> I Wonder if Drill can read data from a RDBMS through a good old JDBC
> driver
> >>
> >> Examples are welcome.
> >>
> >> Best regards
> >>
> >> Juju
> >>
> >>
>
>
Re: JDBC data source
Posted by Chevalier Julien <jc...@numerigraphe.com>.
Yash, what you pointed out is a way to produce Drill output through a JDBC.
I want Drill to consume data from JDBC.
As Carol clearly explains what I need is a JDBC Storage Plugin.
Precisely are they any JDBC Storage plugin available ?
Best regards.
Juju
> Le 6 mars 2015 à 15:18, Yash Sharma <ya...@gmail.com> a écrit :
>
> Hi Juju, check if this can help:
>
> https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL
>
> Regards
>
>
>
> On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien <jc...@numerigraphe.com>
> wrote:
>
>> Dear List
>>
>> I Wonder if Drill can read data from a RDBMS through a good old JDBC driver
>>
>> Examples are welcome.
>>
>> Best regards
>>
>> Juju
>>
>>
Re: JDBC data source
Posted by Yash Sharma <ya...@gmail.com>.
Hi Juju, check if this can help:
https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL
Regards
On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien <jc...@numerigraphe.com>
wrote:
> Dear List
>
> I Wonder if Drill can read data from a RDBMS through a good old JDBC driver
>
> Examples are welcome.
>
> Best regards
>
> Juju
>
>
Re: JDBC data source
Posted by Carol McDonald <cm...@maprtech.com>.
jdbc connects you to a drill bit . drill bits use storage plugins to
connect to data stores. There needs to be a rdbms storage plugin to read
data from a rdbms.
http://drill.apache.org/architecture/
The flow of a Drill query typically involves the following steps:
- The Drill client issues a query. A Drill client is a JDBC, ODBC,
command line interface or a REST API. Any Drillbit in the cluster can
accept queries from the clients. There is no master-slave concept.
- The Drillbit then parses the query, optimizes it, and generates a
distributed query plan that is optimized for fast and efficient execution.
- The Drillbit that accepts the query becomes the driving Drillbit node
for the request. It gets a list of available Drillbit nodes in the cluster
from ZooKeeper. The driving Drillbit determines the appropriate nodes to
execute various query plan fragments to maximize data locality.
- The Drillbit schedules the execution of query fragments on individual
nodes according to the execution plan.
- The individual nodes finish their execution and return data to the
driving Drillbit.
- The driving Drillbit streams results back to the client.
Storage plugin interfaces: Drill serves as a query layer on top of several
data sources. Storage plugins in Drill represent the abstractions that
Drill uses to interact with the data sources. Storage plugins provide Drill
with the following information:
• Metadata available in the source
• Interfaces for Drill to read from and write to data sources
• Location of data and a set of optimization rules to help with efficient
and faster execution of Drill queries on a specific data source
In the context of Hadoop, Drill provides storage plugins for files and
HBase/M7. Drill also integrates with Hive as a storage plugin since Hive
provides a metadata abstraction layer on top of files, HBase/M7, and
provides libraries to read data and operate on these sources (SerDes and
UDFs).
When users query files and HBase/M7 with Drill, they can do it directly or
go through Hive if they have metadata defined there. Drill integration with
Hive is only for metadata. Drill does not invoke the Hive execution engine
for any requests.
On Fri, Mar 6, 2015 at 8:43 AM, Chevalier Julien <jc...@numerigraphe.com>
wrote:
> Dear List
>
> I Wonder if Drill can read data from a RDBMS through a good old JDBC driver
>
> Examples are welcome.
>
> Best regards
>
> Juju
>
>