You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Laurent Goujon <la...@dremio.com> on 2016/08/22 22:24:53 UTC
WIP for prepared statement/metadata querying in C++ native client
Hi,
I just started working on adding support for prepared statements and
metadata querying in the C++ Drill client. Hopefully, nobody else has
started working on this (The Drill jiras don't mention any activity on
this), but if it is not the case, let me know.
My working branch is
https://github.com/laurentgo/drill/tree/laurent/improve-native-client.
For now, I just have a basic interface API (
https://github.com/laurentgo/drill/commit/1f55a3e631cd97016b113b9d4bca07b5e016a25e),
and it would be nice if people knowledgeable about the C++ client could
review it and give me some feedback. I'm also adding an actual initial
implementation in the coming days.
Cheers,
Laurent
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Parth Chandra <pc...@maprtech.com>.
That's along the lines of what I was asking. What I'm hoping for, though,
is a concrete proposal.
As you say, we could be doing a bunch of improvements, but before we start
talking about changing the ODBC driver, I would recommend we have at least
one incremental improvement that end users will benefit from.
On Wed, Aug 24, 2016 at 11:50 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> I'm not sure what unused code you are worried about. The backend already
> implements these APIs and these changes would simply expose that interface
> to the C++ consumers. I'd expect the ODBC driver to be updated to take
> advantage of these APIs and thus all the code would be exercised all the
> time. The main difference is: once we have the new client in user's hands,
> we can iterate on performance improvements without having to upgrade the
> client again in the future. Since this is an additive change, the existing
> clients will continue to work without issue.
>
> I think we already have some jiras about improving metadata performance and
> BI tool experience. I'll try to dig them up. The thought was to start as
> simple as possible, which is what the backend does now: do what the client
> is doing but on the server. The next clear step for metadata preparation is
> to do single parsing & planning (without the limit 0) so that in known
> schema cases, we can avoid double parsing/planning (especially expensive in
> partition pruning and metadata cache cases). Note, I'd expect that this
> next version will still maintain the same behavior if the RowType is not
> known on the initial DrillTable (where we don't have the limit 0 short
> circuit today)
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Aug 23, 2016 at 6:30 PM, Parth Chandra <pa...@apache.org> wrote:
>
> > I would think that the Java client would be sufficient for
> experimentation.
> > But what I'm looking for is an actual proposal for the backend changes.
> If
> > we don't have one (not sure if there is a JIRA open for it), then we
> should
> > start that now.
> > Otherwise I'm afraid we will end up with a bunch of large amount of code
> > that is not used. In particular, for the C++ client, I would like to
> avoid
> > that.
> >
> >
> >
> >
> > On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > The clear quick win would be caching parsing/planning/pruning on the
> > server
> > > and reusing (if executed within time t or until statement/connection
> are
> > > closed).
> > >
> > > My thinking is to get an implementation of the client that is opaque to
> > the
> > > server implementation so that we can iterate on preparation without
> > having
> > > to constantly update the client. From there, developers can easily
> > > experiment with different mechanisms to find what works best for BI
> > tools.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <la...@dremio.com>
> > > wrote:
> > >
> > > > I'm currently focusing on the client work, and making sure the C++
> > > client
> > > > is not lagging behind the Java one. I personally haven't worked on
> > > backend
> > > > changes for prepared statements.
> > > >
> > > > Laurent
> > > >
> > > > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <pa...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Laurent,
> > > > >
> > > > > I'll take a look at this in the next few days.
> > > > >
> > > > > On a related note, do you or Venki have a proposal for the
> backend
> > > > > changes (i.e actual implementation of prepare)? It would be a good
> > idea
> > > > to
> > > > > start a discussion on that.
> > > > >
> > > > > Parth
> > > > >
> > > > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <
> laurent@dremio.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I just started working on adding support for prepared statements
> > and
> > > > > > metadata querying in the C++ Drill client. Hopefully, nobody else
> > has
> > > > > > started working on this (The Drill jiras don't mention any
> activity
> > > on
> > > > > > this), but if it is not the case, let me know.
> > > > > >
> > > > > > My working branch is
> > > > > > https://github.com/laurentgo/drill/tree/laurent/improve-
> > > native-client.
> > > > > >
> > > > > > For now, I just have a basic interface API (
> > > > > > https://github.com/laurentgo/drill/commit/
> > > > 1f55a3e631cd97016b113b9d4bca07
> > > > > > b5e016a25e),
> > > > > > and it would be nice if people knowledgeable about the C++ client
> > > could
> > > > > > review it and give me some feedback. I'm also adding an actual
> > > initial
> > > > > > implementation in the coming days.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Laurent
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Jacques Nadeau <ja...@dremio.com>.
I'm not sure what unused code you are worried about. The backend already
implements these APIs and these changes would simply expose that interface
to the C++ consumers. I'd expect the ODBC driver to be updated to take
advantage of these APIs and thus all the code would be exercised all the
time. The main difference is: once we have the new client in user's hands,
we can iterate on performance improvements without having to upgrade the
client again in the future. Since this is an additive change, the existing
clients will continue to work without issue.
I think we already have some jiras about improving metadata performance and
BI tool experience. I'll try to dig them up. The thought was to start as
simple as possible, which is what the backend does now: do what the client
is doing but on the server. The next clear step for metadata preparation is
to do single parsing & planning (without the limit 0) so that in known
schema cases, we can avoid double parsing/planning (especially expensive in
partition pruning and metadata cache cases). Note, I'd expect that this
next version will still maintain the same behavior if the RowType is not
known on the initial DrillTable (where we don't have the limit 0 short
circuit today)
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Aug 23, 2016 at 6:30 PM, Parth Chandra <pa...@apache.org> wrote:
> I would think that the Java client would be sufficient for experimentation.
> But what I'm looking for is an actual proposal for the backend changes. If
> we don't have one (not sure if there is a JIRA open for it), then we should
> start that now.
> Otherwise I'm afraid we will end up with a bunch of large amount of code
> that is not used. In particular, for the C++ client, I would like to avoid
> that.
>
>
>
>
> On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > The clear quick win would be caching parsing/planning/pruning on the
> server
> > and reusing (if executed within time t or until statement/connection are
> > closed).
> >
> > My thinking is to get an implementation of the client that is opaque to
> the
> > server implementation so that we can iterate on preparation without
> having
> > to constantly update the client. From there, developers can easily
> > experiment with different mechanisms to find what works best for BI
> tools.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <la...@dremio.com>
> > wrote:
> >
> > > I'm currently focusing on the client work, and making sure the C++
> > client
> > > is not lagging behind the Java one. I personally haven't worked on
> > backend
> > > changes for prepared statements.
> > >
> > > Laurent
> > >
> > > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <pa...@apache.org>
> > wrote:
> > >
> > > > Hi Laurent,
> > > >
> > > > I'll take a look at this in the next few days.
> > > >
> > > > On a related note, do you or Venki have a proposal for the backend
> > > > changes (i.e actual implementation of prepare)? It would be a good
> idea
> > > to
> > > > start a discussion on that.
> > > >
> > > > Parth
> > > >
> > > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <la...@dremio.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I just started working on adding support for prepared statements
> and
> > > > > metadata querying in the C++ Drill client. Hopefully, nobody else
> has
> > > > > started working on this (The Drill jiras don't mention any activity
> > on
> > > > > this), but if it is not the case, let me know.
> > > > >
> > > > > My working branch is
> > > > > https://github.com/laurentgo/drill/tree/laurent/improve-
> > native-client.
> > > > >
> > > > > For now, I just have a basic interface API (
> > > > > https://github.com/laurentgo/drill/commit/
> > > 1f55a3e631cd97016b113b9d4bca07
> > > > > b5e016a25e),
> > > > > and it would be nice if people knowledgeable about the C++ client
> > could
> > > > > review it and give me some feedback. I'm also adding an actual
> > initial
> > > > > implementation in the coming days.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Laurent
> > > > >
> > > >
> > >
> >
>
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Parth Chandra <pa...@apache.org>.
I would think that the Java client would be sufficient for experimentation.
But what I'm looking for is an actual proposal for the backend changes. If
we don't have one (not sure if there is a JIRA open for it), then we should
start that now.
Otherwise I'm afraid we will end up with a bunch of large amount of code
that is not used. In particular, for the C++ client, I would like to avoid
that.
On Tue, Aug 23, 2016 at 6:09 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> The clear quick win would be caching parsing/planning/pruning on the server
> and reusing (if executed within time t or until statement/connection are
> closed).
>
> My thinking is to get an implementation of the client that is opaque to the
> server implementation so that we can iterate on preparation without having
> to constantly update the client. From there, developers can easily
> experiment with different mechanisms to find what works best for BI tools.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <la...@dremio.com>
> wrote:
>
> > I'm currently focusing on the client work, and making sure the C++
> client
> > is not lagging behind the Java one. I personally haven't worked on
> backend
> > changes for prepared statements.
> >
> > Laurent
> >
> > On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> > > Hi Laurent,
> > >
> > > I'll take a look at this in the next few days.
> > >
> > > On a related note, do you or Venki have a proposal for the backend
> > > changes (i.e actual implementation of prepare)? It would be a good idea
> > to
> > > start a discussion on that.
> > >
> > > Parth
> > >
> > > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <la...@dremio.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I just started working on adding support for prepared statements and
> > > > metadata querying in the C++ Drill client. Hopefully, nobody else has
> > > > started working on this (The Drill jiras don't mention any activity
> on
> > > > this), but if it is not the case, let me know.
> > > >
> > > > My working branch is
> > > > https://github.com/laurentgo/drill/tree/laurent/improve-
> native-client.
> > > >
> > > > For now, I just have a basic interface API (
> > > > https://github.com/laurentgo/drill/commit/
> > 1f55a3e631cd97016b113b9d4bca07
> > > > b5e016a25e),
> > > > and it would be nice if people knowledgeable about the C++ client
> could
> > > > review it and give me some feedback. I'm also adding an actual
> initial
> > > > implementation in the coming days.
> > > >
> > > > Cheers,
> > > >
> > > > Laurent
> > > >
> > >
> >
>
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Jacques Nadeau <ja...@dremio.com>.
The clear quick win would be caching parsing/planning/pruning on the server
and reusing (if executed within time t or until statement/connection are
closed).
My thinking is to get an implementation of the client that is opaque to the
server implementation so that we can iterate on preparation without having
to constantly update the client. From there, developers can easily
experiment with different mechanisms to find what works best for BI tools.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Aug 23, 2016 at 1:54 PM, Laurent Goujon <la...@dremio.com> wrote:
> I'm currently focusing on the client work, and making sure the C++ client
> is not lagging behind the Java one. I personally haven't worked on backend
> changes for prepared statements.
>
> Laurent
>
> On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <pa...@apache.org> wrote:
>
> > Hi Laurent,
> >
> > I'll take a look at this in the next few days.
> >
> > On a related note, do you or Venki have a proposal for the backend
> > changes (i.e actual implementation of prepare)? It would be a good idea
> to
> > start a discussion on that.
> >
> > Parth
> >
> > On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <la...@dremio.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I just started working on adding support for prepared statements and
> > > metadata querying in the C++ Drill client. Hopefully, nobody else has
> > > started working on this (The Drill jiras don't mention any activity on
> > > this), but if it is not the case, let me know.
> > >
> > > My working branch is
> > > https://github.com/laurentgo/drill/tree/laurent/improve-native-client.
> > >
> > > For now, I just have a basic interface API (
> > > https://github.com/laurentgo/drill/commit/
> 1f55a3e631cd97016b113b9d4bca07
> > > b5e016a25e),
> > > and it would be nice if people knowledgeable about the C++ client could
> > > review it and give me some feedback. I'm also adding an actual initial
> > > implementation in the coming days.
> > >
> > > Cheers,
> > >
> > > Laurent
> > >
> >
>
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Laurent Goujon <la...@dremio.com>.
I'm currently focusing on the client work, and making sure the C++ client
is not lagging behind the Java one. I personally haven't worked on backend
changes for prepared statements.
Laurent
On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra <pa...@apache.org> wrote:
> Hi Laurent,
>
> I'll take a look at this in the next few days.
>
> On a related note, do you or Venki have a proposal for the backend
> changes (i.e actual implementation of prepare)? It would be a good idea to
> start a discussion on that.
>
> Parth
>
> On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <la...@dremio.com>
> wrote:
>
> > Hi,
> >
> > I just started working on adding support for prepared statements and
> > metadata querying in the C++ Drill client. Hopefully, nobody else has
> > started working on this (The Drill jiras don't mention any activity on
> > this), but if it is not the case, let me know.
> >
> > My working branch is
> > https://github.com/laurentgo/drill/tree/laurent/improve-native-client.
> >
> > For now, I just have a basic interface API (
> > https://github.com/laurentgo/drill/commit/1f55a3e631cd97016b113b9d4bca07
> > b5e016a25e),
> > and it would be nice if people knowledgeable about the C++ client could
> > review it and give me some feedback. I'm also adding an actual initial
> > implementation in the coming days.
> >
> > Cheers,
> >
> > Laurent
> >
>
Re: WIP for prepared statement/metadata querying in C++ native client
Posted by Parth Chandra <pa...@apache.org>.
Hi Laurent,
I'll take a look at this in the next few days.
On a related note, do you or Venki have a proposal for the backend
changes (i.e actual implementation of prepare)? It would be a good idea to
start a discussion on that.
Parth
On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon <la...@dremio.com> wrote:
> Hi,
>
> I just started working on adding support for prepared statements and
> metadata querying in the C++ Drill client. Hopefully, nobody else has
> started working on this (The Drill jiras don't mention any activity on
> this), but if it is not the case, let me know.
>
> My working branch is
> https://github.com/laurentgo/drill/tree/laurent/improve-native-client.
>
> For now, I just have a basic interface API (
> https://github.com/laurentgo/drill/commit/1f55a3e631cd97016b113b9d4bca07
> b5e016a25e),
> and it would be nice if people knowledgeable about the C++ client could
> review it and give me some feedback. I'm also adding an actual initial
> implementation in the coming days.
>
> Cheers,
>
> Laurent
>