You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Damien Deom <da...@energisme.com> on 2022/04/13 10:05:28 UTC
Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Hi,
I'd like to know if Apache Drill allows to easily build an ODBC/JDBC driver that performs RPC calls to api's we're developing in our company.
Solutions like like Progress allows to do that : https://www.progress.com/tutorials/odbc/2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api
Apache Calcites does that too, but does not implement ODBC
https://calcite.apache.org/avatica/docs/index.html
Thanks in advance,
[cid:image001.jpg@01D84F2C.71052EB0]
Damien DEOM
Head of Lab
Mobile : +33 6 28 07 06 41
@ : damien.deom@loamics.com<ma...@loamics.com>
www.loamics.com<http://www.loamics.com/>
Re: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Posted by James Turton <dz...@apache.org>.
At the current time there is unfortunately no open source ODBC driver
for Drill and updates to the MapR closed source driver have stopped. I
believe it does still work, but obviously an open source driver would be
preferable by a long shot.
On 2022/04/22 14:56, Damien Deom wrote:
> Hi James,
>
> Thank you for your answer. I will take a deeper look at this…
>
> At Loamics, we’re dealing with large datasets coming from sevreral databaes.
>
> The challenge is whether we can perform joins with sufficient
> performance in real time. If not, we will have to go through a data
> preparation phase.
>
> Looking closer, I noticed that drill as an http plugin, but does not
> implement join filter pushdown :
>
> https://drill.apache.org/docs/http-storage-plugin/
> <https://drill.apache.org/docs/http-storage-plugin/>
>
> Do you plan to implement this feature ? will I have to create my own
> connector in order to achieve best performance ?
>
> Also, I can see that the .msi ODBC driver has not been updated since
> October 2018 :
>
> http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/
> <http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/>
>
> See you soon, and keep the good work !
>
>
>
> *Damien DEOM*
>
> Head of Lab
>
> Mobile : +33 6 28 07 06 41
>
> @ : damien.deom@loamics.com <ma...@loamics.com>
>
> *www.loamics.com* <http://www.loamics.com/>**
>
> *De :*James Turton <dz...@apache.org>
> *Envoyé :* mercredi 13 avril 2022 15:24
> *À :* user@drill.apache.org; Damien Deom <da...@energisme.com>
> *Objet :* Re: Apache drill: how to build a custom ODBC/JDBC driver that
> performs rest api calls
>
> Hi Damien
>
> This is not quite the same thing but Drill does include a JDBC driver
> for its clients and it can query HTTP APIs (with a couple of caveats)
> through its HTTP storage plugin. The net effect is that without much
> legwork you can query data from an HTTP API by sending SQL statements
> over a JDBC connection to Drill. One of our community, Charles, built
> pretty much all of that functionality and is quite active on here so he
> may be able to guide you further.
>
> James
>
> On 2022/04/13 12:05, Damien Deom wrote:
>
> Hi,
>
> I’d like to know if Apache Drill allows to easily build an
> ODBC/JDBC driver that performs RPC calls to api’s we’re developing
> in our company.
>
> Solutions like like Progress allows to do that :
> https://www.progress.com/tutorials/odbc/2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.progress.com%2ftutorials%2fodbc%2f2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api&c=E,1,Urgj-NuXu90dZgfTsgtEYFo8kQDL4jPOSzf51rvMzGrLrlyJtj7PBbriIvmAzedneFbKbDn5tsvKLzkEvjP9A3G3emncEtvtmZp-nbge7BgiuyX92RZqMvs,&typo=1>
>
> Apache Calcites does that too, but does not implement ODBC
>
> https://calcite.apache.org/avatica/docs/index.html
> <https://calcite.apache.org/avatica/docs/index.html>
>
> Thanks in advance,
>
>
>
> *Damien DEOM*
>
> Head of Lab
>
> Mobile : +33 6 28 07 06 41
>
> @ : damien.deom@loamics.com <ma...@loamics.com>
>
> *www.loamics.com*
> <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.loamics.com%2f&c=E,1,csUnRmHisqLnkrioD06sE5UFvufpfbiKpODYMkS7KJ05LB3IVoyopISz1k7B7uO-Xj7U2YCKUs2MTReN-88o9OemD50LEljF61vSr-WZ&typo=1>
>
Re: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Posted by Charles Givre <cg...@gmail.com>.
Hi Damien,
Thanks for your interest in Drill. I'm one of the main authors of the HTTP plugin, and it does implement filter pushdowns, but not join filters. This is quite tricky to build in a generic way for APIs, but I'd be happy to discuss with you further. We'd certainly welcome the contribution if you're willing.
Best,
-- C
> On Apr 22, 2022, at 8:56 AM, Damien Deom <da...@energisme.com> wrote:
>
> Hi James,
>
> Thank you for your answer. I will take a deeper look at this…
>
> At Loamics, we’re dealing with large datasets coming from sevreral databaes.
> The challenge is whether we can perform joins with sufficient performance in real time. If not, we will have to go through a data preparation phase.
>
> Looking closer, I noticed that drill as an http plugin, but does not implement join filter pushdown :
> https://drill.apache.org/docs/http-storage-plugin/ <https://drill.apache.org/docs/http-storage-plugin/>
> Do you plan to implement this feature ? will I have to create my own connector in order to achieve best performance ?
>
> Also, I can see that the .msi ODBC driver has not been updated since October 2018 :
> http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/ <http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/>
>
> See you soon, and keep the good work !
>
>
> Damien DEOM
> Head of Lab
> Mobile : +33 6 28 07 06 41
> @ : damien.deom@loamics.com <ma...@loamics.com>
> www.loamics.com <http://www.loamics.com/>
>
>
> De : James Turton <dzamo@apache.org <ma...@apache.org>>
> Envoyé : mercredi 13 avril 2022 15:24
> À : user@drill.apache.org <ma...@drill.apache.org>; Damien Deom <damien.deom@energisme.com <ma...@energisme.com>>
> Objet : Re: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
>
> Hi Damien
>
> This is not quite the same thing but Drill does include a JDBC driver for its clients and it can query HTTP APIs (with a couple of caveats) through its HTTP storage plugin. The net effect is that without much legwork you can query data from an HTTP API by sending SQL statements over a JDBC connection to Drill. One of our community, Charles, built pretty much all of that functionality and is quite active on here so he may be able to guide you further.
>
> James
>
> On 2022/04/13 12:05, Damien Deom wrote:
> Hi,
>
>
> I’d like to know if Apache Drill allows to easily build an ODBC/JDBC driver that performs RPC calls to api’s we’re developing in our company.
>
> Solutions like like Progress allows to do that : https://www.progress.com/tutorials/odbc/2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.progress.com%2ftutorials%2fodbc%2f2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api&c=E,1,Urgj-NuXu90dZgfTsgtEYFo8kQDL4jPOSzf51rvMzGrLrlyJtj7PBbriIvmAzedneFbKbDn5tsvKLzkEvjP9A3G3emncEtvtmZp-nbge7BgiuyX92RZqMvs,&typo=1>
>
> Apache Calcites does that too, but does not implement ODBC
> https://calcite.apache.org/avatica/docs/index.html <https://calcite.apache.org/avatica/docs/index.html>
>
> Thanks in advance,
>
>
>
> Damien DEOM
> Head of Lab
> Mobile : +33 6 28 07 06 41
> @ : damien.deom@loamics.com <ma...@loamics.com>
> www.loamics.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.loamics.com%2f&c=E,1,csUnRmHisqLnkrioD06sE5UFvufpfbiKpODYMkS7KJ05LB3IVoyopISz1k7B7uO-Xj7U2YCKUs2MTReN-88o9OemD50LEljF61vSr-WZ&typo=1>
>
>
>
RE: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Posted by Damien Deom <da...@energisme.com>.
Hi James,
Thank you for your answer. I will take a deeper look at this…
At Loamics, we’re dealing with large datasets coming from sevreral databaes.
The challenge is whether we can perform joins with sufficient performance in real time. If not, we will have to go through a data preparation phase.
Looking closer, I noticed that drill as an http plugin, but does not implement join filter pushdown :
https://drill.apache.org/docs/http-storage-plugin/
Do you plan to implement this feature ? will I have to create my own connector in order to achieve best performance ?
Also, I can see that the .msi ODBC driver has not been updated since October 2018 :
http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.5.1.1002/
See you soon, and keep the good work !
[cid:image001.jpg@01D854A0.C9A84BA0]
Damien DEOM
Head of Lab
Mobile : +33 6 28 07 06 41
@ : damien.deom@loamics.com<ma...@loamics.com>
www.loamics.com<http://www.loamics.com/>
De : James Turton <dz...@apache.org>
Envoyé : mercredi 13 avril 2022 15:24
À : user@drill.apache.org; Damien Deom <da...@energisme.com>
Objet : Re: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Hi Damien
This is not quite the same thing but Drill does include a JDBC driver for its clients and it can query HTTP APIs (with a couple of caveats) through its HTTP storage plugin. The net effect is that without much legwork you can query data from an HTTP API by sending SQL statements over a JDBC connection to Drill. One of our community, Charles, built pretty much all of that functionality and is quite active on here so he may be able to guide you further.
James
On 2022/04/13 12:05, Damien Deom wrote:
Hi,
I’d like to know if Apache Drill allows to easily build an ODBC/JDBC driver that performs RPC calls to api’s we’re developing in our company.
Solutions like like Progress allows to do that : https://www.progress.com/tutorials/odbc/2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.progress.com%2ftutorials%2fodbc%2f2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api&c=E,1,Urgj-NuXu90dZgfTsgtEYFo8kQDL4jPOSzf51rvMzGrLrlyJtj7PBbriIvmAzedneFbKbDn5tsvKLzkEvjP9A3G3emncEtvtmZp-nbge7BgiuyX92RZqMvs,&typo=1>
Apache Calcites does that too, but does not implement ODBC
https://calcite.apache.org/avatica/docs/index.html
Thanks in advance,
[cid:image001.jpg@01D84F2C.71052EB0]
Damien DEOM
Head of Lab
Mobile : +33 6 28 07 06 41
@ : damien.deom@loamics.com<ma...@loamics.com>
www.loamics.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.loamics.com%2f&c=E,1,csUnRmHisqLnkrioD06sE5UFvufpfbiKpODYMkS7KJ05LB3IVoyopISz1k7B7uO-Xj7U2YCKUs2MTReN-88o9OemD50LEljF61vSr-WZ&typo=1>
Re: Apache drill: how to build a custom ODBC/JDBC driver that performs rest api calls
Posted by James Turton <dz...@apache.org>.
Hi Damien
This is not quite the same thing but Drill does include a JDBC driver
for its clients and it can query HTTP APIs (with a couple of caveats)
through its HTTP storage plugin. The net effect is that without much
legwork you can query data from an HTTP API by sending SQL statements
over a JDBC connection to Drill. One of our community, Charles, built
pretty much all of that functionality and is quite active on here so he
may be able to guide you further.
James
On 2022/04/13 12:05, Damien Deom wrote:
>
> Hi,
>
> I’d like to know if Apache Drill allows to easily build an ODBC/JDBC
> driver that performs RPC calls to api’s we’re developing in our company.
>
> Solutions like like Progress allows to do that :
> https://www.progress.com/tutorials/odbc/2-hour-tutorial-build-your-own-custom-odbc-driver-for-rest-api
>
> Apache Calcites does that too, but does not implement ODBC
>
> https://calcite.apache.org/avatica/docs/index.html
>
> Thanks in advance,
>
>
>
> *Damien DEOM*
>
> Head of Lab
>
> Mobile : +33 6 28 07 06 41
>
> @ : damien.deom@loamics.com <ma...@loamics.com>
>
> *www.loamics.com <http://www.loamics.com/>*
>