You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andy Grove <an...@gmail.com> on 2020/01/29 15:43:26 UTC
JDBC / Flight questions
We're pretty close to having a Flight server implemented in Rust, executing
SQL queries with DataFusion and I'm testing with a Java Flight client.
Once this is working, I want to build a Flight JDBC driver. I see that we
already have a JDBC driver in the project but I'm struggling to find
documentation about it. I searched all the markdown files in the Java
project and there are no mentions of JDBC. A quick Google search didn't
show anything immediately either.
I want to get involved in the Java implementation and this is a good place
for me to start and maybe my first task should be documenting the current
JDBC driver. Can anyone tell me what the goals are for the current driver?
Are there some docs somewhere that I missed?
Thanks,
Andy.
Re: JDBC / Flight questions
Posted by Andy Grove <an...@gmail.com>.
I filed a JIRA for this work [1]. Please add questions and comments here. I
already started on some PoC code and will be creating a PR in the next 1-2
weeks.
[1] https://issues.apache.org/jira/browse/ARROW-7744
On Wed, Jan 29, 2020 at 10:44 AM Andy Grove <an...@gmail.com> wrote:
> Thanks for all the information. This is very helpful.
>
> Andy.
>
> On Wed, Jan 29, 2020 at 9:24 AM Jacques Nadeau <ja...@apache.org> wrote:
>
>> At Dremio we have two things at the moment:
>>
>> A JDBC driver that is built on Arrow and served as the inspiration for
>> some
>> of the design choices in flight [1]
>> A preview flight connector that doesn't yet expose JDBC [2]
>>
>> These the former is built on Avatica (part of the Apache Calcite project)
>> so the thinking has been that a Avatica on top of Flight client might make
>> the most sense. I don't believe anyone has started thinking through how to
>> do that though.
>>
>> [1] https://github.com/dremio/dremio-oss/tree/master/client/jdbc
>> [2] https://github.com/dremio-hub/dremio-flight-connector
>>
>> On Wed, Jan 29, 2020 at 8:18 AM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>> > Sorry as a follow-up Uwe wrote a blog post [1] about using the JDBC to
>> > Arrow code via python. Other than that I'm not aware of anything.
>> >
>> > Thanks,
>> > Micah
>> >
>> > [1]
>> >
>> >
>> https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
>> >
>> > On Wed, Jan 29, 2020 at 8:07 AM Micah Kornfield <em...@gmail.com>
>> > wrote:
>> >
>> > > Hi Andy,
>> > > As far as I know there is no jdbc driver. What exists today is a
>> wrapper
>> > > to convert jdbc resultset to arrow.
>> > >
>> > > I believe dremio might have open sourced some code in their repo
>> which
>> > > wraps arrow inside of a jdbc driver. not sure if it uses flight or
>> > another
>> > > protocol for actual transport
>> > >
>> > > I think a driver on top of arrow data would be a good contribution.
>> > >
>> > > Thanks,
>> > > Micah
>> > >
>> > > On Wednesday, January 29, 2020, Andy Grove <an...@gmail.com>
>> > wrote:
>> > >
>> > >> We're pretty close to having a Flight server implemented in Rust,
>> > >> executing
>> > >> SQL queries with DataFusion and I'm testing with a Java Flight
>> client.
>> > >>
>> > >> Once this is working, I want to build a Flight JDBC driver. I see
>> that
>> > we
>> > >> already have a JDBC driver in the project but I'm struggling to find
>> > >> documentation about it. I searched all the markdown files in the Java
>> > >> project and there are no mentions of JDBC. A quick Google search
>> didn't
>> > >> show anything immediately either.
>> > >>
>> > >> I want to get involved in the Java implementation and this is a good
>> > place
>> > >> for me to start and maybe my first task should be documenting the
>> > current
>> > >> JDBC driver. Can anyone tell me what the goals are for the current
>> > driver?
>> > >> Are there some docs somewhere that I missed?
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Andy.
>> > >>
>> > >
>> >
>>
>
Re: JDBC / Flight questions
Posted by Andy Grove <an...@gmail.com>.
Thanks for all the information. This is very helpful.
Andy.
On Wed, Jan 29, 2020 at 9:24 AM Jacques Nadeau <ja...@apache.org> wrote:
> At Dremio we have two things at the moment:
>
> A JDBC driver that is built on Arrow and served as the inspiration for some
> of the design choices in flight [1]
> A preview flight connector that doesn't yet expose JDBC [2]
>
> These the former is built on Avatica (part of the Apache Calcite project)
> so the thinking has been that a Avatica on top of Flight client might make
> the most sense. I don't believe anyone has started thinking through how to
> do that though.
>
> [1] https://github.com/dremio/dremio-oss/tree/master/client/jdbc
> [2] https://github.com/dremio-hub/dremio-flight-connector
>
> On Wed, Jan 29, 2020 at 8:18 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Sorry as a follow-up Uwe wrote a blog post [1] about using the JDBC to
> > Arrow code via python. Other than that I'm not aware of anything.
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> >
> https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
> >
> > On Wed, Jan 29, 2020 at 8:07 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > Hi Andy,
> > > As far as I know there is no jdbc driver. What exists today is a
> wrapper
> > > to convert jdbc resultset to arrow.
> > >
> > > I believe dremio might have open sourced some code in their repo which
> > > wraps arrow inside of a jdbc driver. not sure if it uses flight or
> > another
> > > protocol for actual transport
> > >
> > > I think a driver on top of arrow data would be a good contribution.
> > >
> > > Thanks,
> > > Micah
> > >
> > > On Wednesday, January 29, 2020, Andy Grove <an...@gmail.com>
> > wrote:
> > >
> > >> We're pretty close to having a Flight server implemented in Rust,
> > >> executing
> > >> SQL queries with DataFusion and I'm testing with a Java Flight client.
> > >>
> > >> Once this is working, I want to build a Flight JDBC driver. I see that
> > we
> > >> already have a JDBC driver in the project but I'm struggling to find
> > >> documentation about it. I searched all the markdown files in the Java
> > >> project and there are no mentions of JDBC. A quick Google search
> didn't
> > >> show anything immediately either.
> > >>
> > >> I want to get involved in the Java implementation and this is a good
> > place
> > >> for me to start and maybe my first task should be documenting the
> > current
> > >> JDBC driver. Can anyone tell me what the goals are for the current
> > driver?
> > >> Are there some docs somewhere that I missed?
> > >>
> > >> Thanks,
> > >>
> > >> Andy.
> > >>
> > >
> >
>
Re: JDBC / Flight questions
Posted by Jacques Nadeau <ja...@apache.org>.
At Dremio we have two things at the moment:
A JDBC driver that is built on Arrow and served as the inspiration for some
of the design choices in flight [1]
A preview flight connector that doesn't yet expose JDBC [2]
These the former is built on Avatica (part of the Apache Calcite project)
so the thinking has been that a Avatica on top of Flight client might make
the most sense. I don't believe anyone has started thinking through how to
do that though.
[1] https://github.com/dremio/dremio-oss/tree/master/client/jdbc
[2] https://github.com/dremio-hub/dremio-flight-connector
On Wed, Jan 29, 2020 at 8:18 AM Micah Kornfield <em...@gmail.com>
wrote:
> Sorry as a follow-up Uwe wrote a blog post [1] about using the JDBC to
> Arrow code via python. Other than that I'm not aware of anything.
>
> Thanks,
> Micah
>
> [1]
>
> https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
>
> On Wed, Jan 29, 2020 at 8:07 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Andy,
> > As far as I know there is no jdbc driver. What exists today is a wrapper
> > to convert jdbc resultset to arrow.
> >
> > I believe dremio might have open sourced some code in their repo which
> > wraps arrow inside of a jdbc driver. not sure if it uses flight or
> another
> > protocol for actual transport
> >
> > I think a driver on top of arrow data would be a good contribution.
> >
> > Thanks,
> > Micah
> >
> > On Wednesday, January 29, 2020, Andy Grove <an...@gmail.com>
> wrote:
> >
> >> We're pretty close to having a Flight server implemented in Rust,
> >> executing
> >> SQL queries with DataFusion and I'm testing with a Java Flight client.
> >>
> >> Once this is working, I want to build a Flight JDBC driver. I see that
> we
> >> already have a JDBC driver in the project but I'm struggling to find
> >> documentation about it. I searched all the markdown files in the Java
> >> project and there are no mentions of JDBC. A quick Google search didn't
> >> show anything immediately either.
> >>
> >> I want to get involved in the Java implementation and this is a good
> place
> >> for me to start and maybe my first task should be documenting the
> current
> >> JDBC driver. Can anyone tell me what the goals are for the current
> driver?
> >> Are there some docs somewhere that I missed?
> >>
> >> Thanks,
> >>
> >> Andy.
> >>
> >
>
Re: JDBC / Flight questions
Posted by Micah Kornfield <em...@gmail.com>.
Sorry as a follow-up Uwe wrote a blog post [1] about using the JDBC to
Arrow code via python. Other than that I'm not aware of anything.
Thanks,
Micah
[1]
https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html
On Wed, Jan 29, 2020 at 8:07 AM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Andy,
> As far as I know there is no jdbc driver. What exists today is a wrapper
> to convert jdbc resultset to arrow.
>
> I believe dremio might have open sourced some code in their repo which
> wraps arrow inside of a jdbc driver. not sure if it uses flight or another
> protocol for actual transport
>
> I think a driver on top of arrow data would be a good contribution.
>
> Thanks,
> Micah
>
> On Wednesday, January 29, 2020, Andy Grove <an...@gmail.com> wrote:
>
>> We're pretty close to having a Flight server implemented in Rust,
>> executing
>> SQL queries with DataFusion and I'm testing with a Java Flight client.
>>
>> Once this is working, I want to build a Flight JDBC driver. I see that we
>> already have a JDBC driver in the project but I'm struggling to find
>> documentation about it. I searched all the markdown files in the Java
>> project and there are no mentions of JDBC. A quick Google search didn't
>> show anything immediately either.
>>
>> I want to get involved in the Java implementation and this is a good place
>> for me to start and maybe my first task should be documenting the current
>> JDBC driver. Can anyone tell me what the goals are for the current driver?
>> Are there some docs somewhere that I missed?
>>
>> Thanks,
>>
>> Andy.
>>
>
Re: JDBC / Flight questions
Posted by Micah Kornfield <em...@gmail.com>.
Hi Andy,
As far as I know there is no jdbc driver. What exists today is a wrapper
to convert jdbc resultset to arrow.
I believe dremio might have open sourced some code in their repo which
wraps arrow inside of a jdbc driver. not sure if it uses flight or another
protocol for actual transport
I think a driver on top of arrow data would be a good contribution.
Thanks,
Micah
On Wednesday, January 29, 2020, Andy Grove <an...@gmail.com> wrote:
> We're pretty close to having a Flight server implemented in Rust, executing
> SQL queries with DataFusion and I'm testing with a Java Flight client.
>
> Once this is working, I want to build a Flight JDBC driver. I see that we
> already have a JDBC driver in the project but I'm struggling to find
> documentation about it. I searched all the markdown files in the Java
> project and there are no mentions of JDBC. A quick Google search didn't
> show anything immediately either.
>
> I want to get involved in the Java implementation and this is a good place
> for me to start and maybe my first task should be documenting the current
> JDBC driver. Can anyone tell me what the goals are for the current driver?
> Are there some docs somewhere that I missed?
>
> Thanks,
>
> Andy.
>