You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Charles Givre <cg...@gmail.com> on 2022/01/03 16:00:25 UTC
[DISCUSS] Restarting the Arrow Conversation
Hello all,
There was a discussion in a recently closed PR [1] with a discussion between z0ltrix, James Turton and a few others about integrating Drill with Apache Arrow and wondering why it was never done. I'd like to share my perspective as someone who has been around Drill for some time but also as someone who never worked for MapR or Dremio. This just represents my understanding of events as an outsider, and I could be wrong about some or all of this. Please forgive (or correct) any inaccuracies.
When I first learned of Arrow and the idea of integrating Arrow with Drill, the thing that interested me the most was the ability to move data between platforms without having to serialize/deserialize the data. From my understanding, MapR did some research and didn't find a significant performance advantage and hence didn't really pursue the integration. The other side of it was that it would require a significant amount of work to refactor major parts of Drill.
I don't know the internal politics, but this was one of the major points of diversion between Dremio and Drill.
With that said, there was a renewed discussion on the list [2] where Paul Rogers proposed what he described as a "Crude but Effective" approach to an Arrow integration.
This is in the email link but here was a part of Paul's email:
> Charles, just brainstorming a bit, I think the easiest way to start is to create a simple, stand-alone server that speaks Arrow to the client, and uses the native Drill client to speak to Drill. The native Drill client exposes Drill value vectors. One trick would be to convert Drill vectors to the Arrow format. I think that data vectors are the same format. Possibly offset vectors. I think Arrow went its own way with null-value (Drill's is-set) vectors. So, some conversion might be a no-op, others might need to rewrite a vector. Good thing, this is purely at the vector level, so would be easy to write. The next issue is the one that Parth has long pointed out: Drill and Arrow each have their own memory allocators. How could we share a data vector between the two? The simplest initial solution is just to copy the data from Drill to Arrow. Slow, but transparent to the client. A crude first-approximation of the development steps:
>
> A crude first-approximation of the development steps:
> 1. Create the client shell server.
> 2. Implement the Arrow client protocol. Need some way to accept a query and return batches of results.
> 3. Forward the query to Drill using the native Drill client.
> 4. As a first pass, copy vectors from Drill to Arrow and return them to the client.
> 5. Then, solve that memory allocator problem to pass data without copying.
One point that Paul made was that these pieces are fairly discrete and could be implemented without refactoring major components of Drill. Of course, this could be something for Drill 2.0. At a minimum, could we take the conversation off of the PR and put it in the email list? ;-)
Let's discuss... All ideas are welcome!
Best,
-- C
[1]: https://github.com/apache/drill/pull/2412 <https://github.com/apache/drill/pull/2412>
[2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l <https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
As a little bit of perspective from somebody who *was* at MapR at the time,
here are my recollections.
Arrow is pretty much the value vectors from Drill with some lessons learned
and all dependencies removed so that Arrow can be consumed separately from
Drill.
The spinout of the Dremio team didn't happen because of the lack of
integration with Arrow ... it was more the other way around ... because a
significant chunk of the Drill team left to form Dremio, the driving force
that could have pushed for integration just wasn't around any more because
they were off doing Dremio and weren't working on Drill any more very much.
The motive for the spinout had mostly to do with the fact that Tomer and
Jacques recognized the opportunity to build a largely in-memory analytical
engine based on zero serialization techniques and also recognized that this
could never be a priority for MapR because it was outside the center of
mass there. Once the Dremio team was out, though, they had a huge need for
interoperability with systems like Spark and Cassandra, and they needed to
not impose all of Drill as a dependency if they wanted these other systems
to take on Arrow.
This history doesn't really impact the merits or methods of integrating
present-day Drill with Arrow, but it is nice to get the story the right way
around.
On Mon, Jan 3, 2022 at 8:00 AM Charles Givre <cg...@gmail.com> wrote:
> Hello all,
> There was a discussion in a recently closed PR [1] with a discussion
> between z0ltrix, James Turton and a few others about integrating Drill with
> Apache Arrow and wondering why it was never done. I'd like to share my
> perspective as someone who has been around Drill for some time but also as
> someone who never worked for MapR or Dremio. This just represents my
> understanding of events as an outsider, and I could be wrong about some or
> all of this. Please forgive (or correct) any inaccuracies.
>
> When I first learned of Arrow and the idea of integrating Arrow with
> Drill, the thing that interested me the most was the ability to move data
> between platforms without having to serialize/deserialize the data. From
> my understanding, MapR did some research and didn't find a significant
> performance advantage and hence didn't really pursue the integration. The
> other side of it was that it would require a significant amount of work to
> refactor major parts of Drill.
>
> I don't know the internal politics, but this was one of the major points
> of diversion between Dremio and Drill.
>
> With that said, there was a renewed discussion on the list [2] where Paul
> Rogers proposed what he described as a "Crude but Effective" approach to an
> Arrow integration.
>
> This is in the email link but here was a part of Paul's email:
>
> > Charles, just brainstorming a bit, I think the easiest way to start is
> to create a simple, stand-alone server that speaks Arrow to the client, and
> uses the native Drill client to speak to Drill. The native Drill client
> exposes Drill value vectors. One trick would be to convert Drill vectors to
> the Arrow format. I think that data vectors are the same format. Possibly
> offset vectors. I think Arrow went its own way with null-value (Drill's
> is-set) vectors. So, some conversion might be a no-op, others might need to
> rewrite a vector. Good thing, this is purely at the vector level, so would
> be easy to write. The next issue is the one that Parth has long pointed
> out: Drill and Arrow each have their own memory allocators. How could we
> share a data vector between the two? The simplest initial solution is just
> to copy the data from Drill to Arrow. Slow, but transparent to the client.
> A crude first-approximation of the development steps:
> >
> > A crude first-approximation of the development steps:
> > 1. Create the client shell server.
> > 2. Implement the Arrow client protocol. Need some way to accept a query
> and return batches of results.
> > 3. Forward the query to Drill using the native Drill client.
> > 4. As a first pass, copy vectors from Drill to Arrow and return them to
> the client.
> > 5. Then, solve that memory allocator problem to pass data without
> copying.
>
> One point that Paul made was that these pieces are fairly discrete and
> could be implemented without refactoring major components of Drill. Of
> course, this could be something for Drill 2.0. At a minimum, could we take
> the conversation off of the PR and put it in the email list? ;-)
>
> Let's discuss... All ideas are welcome!
>
> Best,
> -- C
>
>
> [1]: https://github.com/apache/drill/pull/2412 <
> https://github.com/apache/drill/pull/2412>
> [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l <
> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
>
>
>
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by James Turton <dz...@apache.org>.
GitHub comments do get partial points for the editing requirement. You
can edit them including, I believe, with strikethrough for an audit
trail. But doing that a lot could well leave behind something quite
incoherent while a document would admit reorganisations.
We could also have both? A GitHub issue for while people are in
discussion mode and want to respond to points made by others (using rich
formatting not available here) which gets reduced to a wiki page by some
good soul once most of the dust has settled. The wiki page then
receives ongoing editing from the community like any other wiki page.
I'm happy with whatever folks find valuable and I'm also happy to do
editing work if we land on something that requires that from someone.
On 2022/01/04 21:14, Ted Dunning wrote:
>
> Exactly. I very much had in mind an "On the other hand" kind of document.
>
> The super benefit of a non-threaded presentation is that if I advocate
> something stupid due to an oversight on my part, I can go back and edit
> away the stupid statement (since it shouldn't be part of the consensus)
> and tag anybody who might have responded. I might even leave a note
> saying "You might think X, but that isn't so because of Y" to help later
> readers.
>
> That is all very, very hard to do in threaded discussions.
>
>
>
> On Tue, Jan 4, 2022 at 9:37 AM James Turton <dzamo@apache.org
> <ma...@apache.org>> wrote:
>
> Ah, and I see now that you said as much already. So a collaboratively
> edited document? Wiki pages containing a variety of independent views
> might turn out something like this collection I suppose
>
> https://wiki.c2.com/?GarbageCollection
> <https://wiki.c2.com/?GarbageCollection>
>
> which isn't bad IMHO.
>
> On 2022/01/04 16:42, Ted Dunning wrote:
> > Threading is exactly what I would want to avoid.
> >
> >
> >
> > On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
> <ma...@apache.org>
> > <mailto:dzamo@apache.org <ma...@apache.org>>> wrote:
> >
> > Hi all
> >
> > GitHub Issues allow a conversation thread with rich
> formatting so I
> > propose that we use them for meaty topics like this. Please
> use the
> > "Feature Request" issue template for this purpose, and set
> the issue's
> > Project field to "Drill 2.0"[1], said project having recently
> been
> > created by Charles. I am busy transcribing the current
> discussion from
> > the mailing list and a GitHub PR to just such a new feature
> request at
> >
> > https://github.com/apache/drill/issues/2421
> <https://github.com/apache/drill/issues/2421>
> > <https://github.com/apache/drill/issues/2421
> <https://github.com/apache/drill/issues/2421>>
> >
> > James
> >
> > [1] https://github.com/apache/drill/projects/1
> <https://github.com/apache/drill/projects/1>
> > <https://github.com/apache/drill/projects/1
> <https://github.com/apache/drill/projects/1>>
> >
> > On 2022/01/04 09:49, Ted Dunning wrote:
> > > I wonder if there isn't a better place for this discussion?
> > >
> > > As you point out, there are many threads and many of the
> points
> > are rather
> > > contentious technically. That will make them even harder to
> > follow in an
> > > email thread.
> > >
> > > We could just use the wiki and format the text in the form of
> > questions
> > > with alternative positions.
> > >
> > > Or we could use an open google document with similar form.
> > >
> > > What's the preference here?
> > >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
Paul,
That could be made to work. I suspect that the flow won't be as simple as
with an open google doc having worked in both styles.
I will go either way.
On Tue, Jan 4, 2022 at 5:29 PM Paul Rogers <pa...@gmail.com> wrote:
> Hi Ted,
>
> I like where you're going with how to manage the discussion.
>
> Here's a trick that I saw someone do recently. The design/discussion as a
> PR.
> Comments are just code review comments, tagged to a specific line. The "er,
> never mind"
> aspect that Ted talks about is handled by pushing a new version of the doc
> (if the doc contains the error) or editing a comment (if the comment had
> the
> error.) The history of all changes is in the commit history.
>
> As we go off on tangents (Arrow-based API? Modern way to do code gen?),
> these can
> be handled as new documents.
>
> All we need is a place to put this stuff. A "docs" or "design" directory
> within the
> source tree?
>
> Thanks,
>
> - Paul
>
> On Tue, Jan 4, 2022 at 11:15 AM Ted Dunning <te...@gmail.com> wrote:
>
> > Exactly. I very much had in mind an "On the other hand" kind of document.
> >
> > The super benefit of a non-threaded presentation is that if I advocate
> > something stupid due to an oversight on my part, I can go back and edit
> > away the stupid statement (since it shouldn't be part of the consensus)
> and
> > tag anybody who might have responded. I might even leave a note saying
> "You
> > might think X, but that isn't so because of Y" to help later readers.
> >
> > That is all very, very hard to do in threaded discussions.
> >
> >
> >
> > On Tue, Jan 4, 2022 at 9:37 AM James Turton <dz...@apache.org> wrote:
> >
> > > Ah, and I see now that you said as much already. So a collaboratively
> > > edited document? Wiki pages containing a variety of independent views
> > > might turn out something like this collection I suppose
> > >
> > > https://wiki.c2.com/?GarbageCollection
> > >
> > > which isn't bad IMHO.
> > >
> > > On 2022/01/04 16:42, Ted Dunning wrote:
> > > > Threading is exactly what I would want to avoid.
> > > >
> > > >
> > > >
> > > > On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
> > > > <ma...@apache.org>> wrote:
> > > >
> > > > Hi all
> > > >
> > > > GitHub Issues allow a conversation thread with rich formatting
> so I
> > > > propose that we use them for meaty topics like this. Please use
> > the
> > > > "Feature Request" issue template for this purpose, and set the
> > > issue's
> > > > Project field to "Drill 2.0"[1], said project having recently
> been
> > > > created by Charles. I am busy transcribing the current
> discussion
> > > from
> > > > the mailing list and a GitHub PR to just such a new feature
> request
> > > at
> > > >
> > > > https://github.com/apache/drill/issues/2421
> > > > <https://github.com/apache/drill/issues/2421>
> > > >
> > > > James
> > > >
> > > > [1] https://github.com/apache/drill/projects/1
> > > > <https://github.com/apache/drill/projects/1>
> > > >
> > > > On 2022/01/04 09:49, Ted Dunning wrote:
> > > > > I wonder if there isn't a better place for this discussion?
> > > > >
> > > > > As you point out, there are many threads and many of the
> points
> > > > are rather
> > > > > contentious technically. That will make them even harder to
> > > > follow in an
> > > > > email thread.
> > > > >
> > > > > We could just use the wiki and format the text in the form of
> > > > questions
> > > > > with alternative positions.
> > > > >
> > > > > Or we could use an open google document with similar form.
> > > > >
> > > > > What's the preference here?
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by James Turton <dz...@apache.org>.
There is the docs/dev
<https://github.com/apache/drill/tree/master/docs/dev> directory in the
source tree which might be suitable.
On 2022/01/05 03:29, Paul Rogers wrote:
> Hi Ted,
>
> I like where you're going with how to manage the discussion.
>
> Here's a trick that I saw someone do recently. The design/discussion as a
> PR.
> Comments are just code review comments, tagged to a specific line. The "er,
> never mind"
> aspect that Ted talks about is handled by pushing a new version of the doc
> (if the doc contains the error) or editing a comment (if the comment had the
> error.) The history of all changes is in the commit history.
>
> As we go off on tangents (Arrow-based API? Modern way to do code gen?),
> these can
> be handled as new documents.
>
> All we need is a place to put this stuff. A "docs" or "design" directory
> within the
> source tree?
>
> Thanks,
>
> - Paul
>
> On Tue, Jan 4, 2022 at 11:15 AM Ted Dunning<te...@gmail.com> wrote:
>
>> Exactly. I very much had in mind an "On the other hand" kind of document.
>>
>> The super benefit of a non-threaded presentation is that if I advocate
>> something stupid due to an oversight on my part, I can go back and edit
>> away the stupid statement (since it shouldn't be part of the consensus) and
>> tag anybody who might have responded. I might even leave a note saying "You
>> might think X, but that isn't so because of Y" to help later readers.
>>
>> That is all very, very hard to do in threaded discussions.
>>
>>
>>
>> On Tue, Jan 4, 2022 at 9:37 AM James Turton<dz...@apache.org> wrote:
>>
>>> Ah, and I see now that you said as much already. So a collaboratively
>>> edited document? Wiki pages containing a variety of independent views
>>> might turn out something like this collection I suppose
>>>
>>> https://wiki.c2.com/?GarbageCollection
>>>
>>> which isn't bad IMHO.
>>>
>>> On 2022/01/04 16:42, Ted Dunning wrote:
>>>> Threading is exactly what I would want to avoid.
>>>>
>>>>
>>>>
>>>> On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
>>>> <ma...@apache.org>> wrote:
>>>>
>>>> Hi all
>>>>
>>>> GitHub Issues allow a conversation thread with rich formatting so I
>>>> propose that we use them for meaty topics like this. Please use
>> the
>>>> "Feature Request" issue template for this purpose, and set the
>>> issue's
>>>> Project field to "Drill 2.0"[1], said project having recently been
>>>> created by Charles. I am busy transcribing the current discussion
>>> from
>>>> the mailing list and a GitHub PR to just such a new feature request
>>> at
>>>> https://github.com/apache/drill/issues/2421
>>>> <https://github.com/apache/drill/issues/2421>
>>>>
>>>> James
>>>>
>>>> [1]https://github.com/apache/drill/projects/1
>>>> <https://github.com/apache/drill/projects/1>
>>>>
>>>> On 2022/01/04 09:49, Ted Dunning wrote:
>>>> > I wonder if there isn't a better place for this discussion?
>>>> >
>>>> > As you point out, there are many threads and many of the points
>>>> are rather
>>>> > contentious technically. That will make them even harder to
>>>> follow in an
>>>> > email thread.
>>>> >
>>>> > We could just use the wiki and format the text in the form of
>>>> questions
>>>> > with alternative positions.
>>>> >
>>>> > Or we could use an open google document with similar form.
>>>> >
>>>> > What's the preference here?
>>>> >
>>>>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Paul Rogers <pa...@gmail.com>.
Hi Ted,
I like where you're going with how to manage the discussion.
Here's a trick that I saw someone do recently. The design/discussion as a
PR.
Comments are just code review comments, tagged to a specific line. The "er,
never mind"
aspect that Ted talks about is handled by pushing a new version of the doc
(if the doc contains the error) or editing a comment (if the comment had the
error.) The history of all changes is in the commit history.
As we go off on tangents (Arrow-based API? Modern way to do code gen?),
these can
be handled as new documents.
All we need is a place to put this stuff. A "docs" or "design" directory
within the
source tree?
Thanks,
- Paul
On Tue, Jan 4, 2022 at 11:15 AM Ted Dunning <te...@gmail.com> wrote:
> Exactly. I very much had in mind an "On the other hand" kind of document.
>
> The super benefit of a non-threaded presentation is that if I advocate
> something stupid due to an oversight on my part, I can go back and edit
> away the stupid statement (since it shouldn't be part of the consensus) and
> tag anybody who might have responded. I might even leave a note saying "You
> might think X, but that isn't so because of Y" to help later readers.
>
> That is all very, very hard to do in threaded discussions.
>
>
>
> On Tue, Jan 4, 2022 at 9:37 AM James Turton <dz...@apache.org> wrote:
>
> > Ah, and I see now that you said as much already. So a collaboratively
> > edited document? Wiki pages containing a variety of independent views
> > might turn out something like this collection I suppose
> >
> > https://wiki.c2.com/?GarbageCollection
> >
> > which isn't bad IMHO.
> >
> > On 2022/01/04 16:42, Ted Dunning wrote:
> > > Threading is exactly what I would want to avoid.
> > >
> > >
> > >
> > > On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
> > > <ma...@apache.org>> wrote:
> > >
> > > Hi all
> > >
> > > GitHub Issues allow a conversation thread with rich formatting so I
> > > propose that we use them for meaty topics like this. Please use
> the
> > > "Feature Request" issue template for this purpose, and set the
> > issue's
> > > Project field to "Drill 2.0"[1], said project having recently been
> > > created by Charles. I am busy transcribing the current discussion
> > from
> > > the mailing list and a GitHub PR to just such a new feature request
> > at
> > >
> > > https://github.com/apache/drill/issues/2421
> > > <https://github.com/apache/drill/issues/2421>
> > >
> > > James
> > >
> > > [1] https://github.com/apache/drill/projects/1
> > > <https://github.com/apache/drill/projects/1>
> > >
> > > On 2022/01/04 09:49, Ted Dunning wrote:
> > > > I wonder if there isn't a better place for this discussion?
> > > >
> > > > As you point out, there are many threads and many of the points
> > > are rather
> > > > contentious technically. That will make them even harder to
> > > follow in an
> > > > email thread.
> > > >
> > > > We could just use the wiki and format the text in the form of
> > > questions
> > > > with alternative positions.
> > > >
> > > > Or we could use an open google document with similar form.
> > > >
> > > > What's the preference here?
> > > >
> > >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
Exactly. I very much had in mind an "On the other hand" kind of document.
The super benefit of a non-threaded presentation is that if I advocate
something stupid due to an oversight on my part, I can go back and edit
away the stupid statement (since it shouldn't be part of the consensus) and
tag anybody who might have responded. I might even leave a note saying "You
might think X, but that isn't so because of Y" to help later readers.
That is all very, very hard to do in threaded discussions.
On Tue, Jan 4, 2022 at 9:37 AM James Turton <dz...@apache.org> wrote:
> Ah, and I see now that you said as much already. So a collaboratively
> edited document? Wiki pages containing a variety of independent views
> might turn out something like this collection I suppose
>
> https://wiki.c2.com/?GarbageCollection
>
> which isn't bad IMHO.
>
> On 2022/01/04 16:42, Ted Dunning wrote:
> > Threading is exactly what I would want to avoid.
> >
> >
> >
> > On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi all
> >
> > GitHub Issues allow a conversation thread with rich formatting so I
> > propose that we use them for meaty topics like this. Please use the
> > "Feature Request" issue template for this purpose, and set the
> issue's
> > Project field to "Drill 2.0"[1], said project having recently been
> > created by Charles. I am busy transcribing the current discussion
> from
> > the mailing list and a GitHub PR to just such a new feature request
> at
> >
> > https://github.com/apache/drill/issues/2421
> > <https://github.com/apache/drill/issues/2421>
> >
> > James
> >
> > [1] https://github.com/apache/drill/projects/1
> > <https://github.com/apache/drill/projects/1>
> >
> > On 2022/01/04 09:49, Ted Dunning wrote:
> > > I wonder if there isn't a better place for this discussion?
> > >
> > > As you point out, there are many threads and many of the points
> > are rather
> > > contentious technically. That will make them even harder to
> > follow in an
> > > email thread.
> > >
> > > We could just use the wiki and format the text in the form of
> > questions
> > > with alternative positions.
> > >
> > > Or we could use an open google document with similar form.
> > >
> > > What's the preference here?
> > >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by James Turton <dz...@apache.org>.
Ah, and I see now that you said as much already. So a collaboratively
edited document? Wiki pages containing a variety of independent views
might turn out something like this collection I suppose
https://wiki.c2.com/?GarbageCollection
which isn't bad IMHO.
On 2022/01/04 16:42, Ted Dunning wrote:
> Threading is exactly what I would want to avoid.
>
>
>
> On Tue, Jan 4, 2022, 3:58 AM James Turton <dzamo@apache.org
> <ma...@apache.org>> wrote:
>
> Hi all
>
> GitHub Issues allow a conversation thread with rich formatting so I
> propose that we use them for meaty topics like this. Please use the
> "Feature Request" issue template for this purpose, and set the issue's
> Project field to "Drill 2.0"[1], said project having recently been
> created by Charles. I am busy transcribing the current discussion from
> the mailing list and a GitHub PR to just such a new feature request at
>
> https://github.com/apache/drill/issues/2421
> <https://github.com/apache/drill/issues/2421>
>
> James
>
> [1] https://github.com/apache/drill/projects/1
> <https://github.com/apache/drill/projects/1>
>
> On 2022/01/04 09:49, Ted Dunning wrote:
> > I wonder if there isn't a better place for this discussion?
> >
> > As you point out, there are many threads and many of the points
> are rather
> > contentious technically. That will make them even harder to
> follow in an
> > email thread.
> >
> > We could just use the wiki and format the text in the form of
> questions
> > with alternative positions.
> >
> > Or we could use an open google document with similar form.
> >
> > What's the preference here?
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
Threading is exactly what I would want to avoid.
On Tue, Jan 4, 2022, 3:58 AM James Turton <dz...@apache.org> wrote:
> Hi all
>
> GitHub Issues allow a conversation thread with rich formatting so I
> propose that we use them for meaty topics like this. Please use the
> "Feature Request" issue template for this purpose, and set the issue's
> Project field to "Drill 2.0"[1], said project having recently been
> created by Charles. I am busy transcribing the current discussion from
> the mailing list and a GitHub PR to just such a new feature request at
>
> https://github.com/apache/drill/issues/2421
>
> James
>
> [1] https://github.com/apache/drill/projects/1
>
> On 2022/01/04 09:49, Ted Dunning wrote:
> > I wonder if there isn't a better place for this discussion?
> >
> > As you point out, there are many threads and many of the points are
> rather
> > contentious technically. That will make them even harder to follow in an
> > email thread.
> >
> > We could just use the wiki and format the text in the form of questions
> > with alternative positions.
> >
> > Or we could use an open google document with similar form.
> >
> > What's the preference here?
> >
>
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by James Turton <dz...@apache.org>.
Hi all
GitHub Issues allow a conversation thread with rich formatting so I
propose that we use them for meaty topics like this. Please use the
"Feature Request" issue template for this purpose, and set the issue's
Project field to "Drill 2.0"[1], said project having recently been
created by Charles. I am busy transcribing the current discussion from
the mailing list and a GitHub PR to just such a new feature request at
https://github.com/apache/drill/issues/2421
James
[1] https://github.com/apache/drill/projects/1
On 2022/01/04 09:49, Ted Dunning wrote:
> I wonder if there isn't a better place for this discussion?
>
> As you point out, there are many threads and many of the points are rather
> contentious technically. That will make them even harder to follow in an
> email thread.
>
> We could just use the wiki and format the text in the form of questions
> with alternative positions.
>
> Or we could use an open google document with similar form.
>
> What's the preference here?
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
I wonder if there isn't a better place for this discussion?
As you point out, there are many threads and many of the points are rather
contentious technically. That will make them even harder to follow in an
email thread.
We could just use the wiki and format the text in the form of questions
with alternative positions.
Or we could use an open google document with similar form.
What's the preference here?
On Mon, Jan 3, 2022 at 7:34 PM Paul Rogers <pa...@gmail.com> wrote:
> Hi Charles,
>
> The material is rather dense and benefits from the Github formatting. To
> preserve it, perhaps we can copy it to a subpage of the Drill 2.0 wiki
> page.
>
> For now, the link to the discussion is [1]. Since the Wiki is not good for
> discussions, let's have that discussion here (if anyone is up to tackling
> such a weighty subject.)
>
> Thanks,
>
> - Paul
>
> [1] https://github.com/apache/drill/pull/2412
>
> On Mon, Jan 3, 2022 at 5:15 PM Charles Givre <cg...@gmail.com> wrote:
>
> > @Paul,
> > Do you mind if I copy the contents of your response to DRILL-8088 to this
> > thread? There's a lot of good info there, and I'd hate to see it get
> lost.
> > -- C
> >
> > > On Jan 3, 2022, at 7:41 PM, Paul Rogers <pa...@gmail.com> wrote:
> > >
> > > Hi All,
> > >
> > > Thanks Charles for dredging up that old discussion, your memory is
> better
> > > than mine! And, thanks Ted for that summary of MapR history. As one of
> > the
> > > "replacement crew" brought in after the original folks left, your
> > > description is consistent with my memory of events. Moreover, as we
> > looked
> > > at what was needed to run Drill in production, an Arrow port was far
> down
> > > on the list: it would not have solved actual customer problems.
> > >
> > > Before we get too excited about Arrow, I think we should have a
> > discussion
> > > about what we want in an internal storage format. I added a long
> (sorry)
> > > set of comments in that PR that Charles mentioned that tries to debunk
> > the
> > > myths that have grown up around using a columnar format as the internal
> > > representation for a query engine. (Columnar is great for storage.) The
> > > note presents the many issues we've encountered over the years that
> have
> > > caused us to layer ever more code on top of vectors to solve various
> > > problems. It also highlights a distributed-systems problem which
> vectors
> > > make far worse.
> > >
> > > Arrow is meant to be portable, as Ted discussed, but it is still
> > columnar,
> > > and this is the source of endless problems in an execution engine. So,
> we
> > > want to ask, what is the optimal format for what Drill actually does?
> I'm
> > > now of the opinion that Drill might actually better benefit from a
> > > row-based format, similar to what Impala uses. The notes even paint a
> > path
> > > forward.
> > >
> > > Ted's description of the goal for Demio suggests that Arrow might be
> the
> > > right answer for that market. Drill, however, tends to be used to query
> > > myriad data sources at scale and as a "query integrator" across
> systems.
> > > This use case has different needs, which may be better served with a
> > > row-based format.
> > >
> > > The upshot is that "value vectors vs. Arrow" is the wrong place to
> start
> > > the discussion. The right place is "what does our many years of
> > experience
> > > with Drill suggest is the most efficient format for how Drill is
> actually
> > > used?"
> > >
> > > Note that Drill could have an Arrow-based API independent of the
> internal
> > > format. The quote from Charles explains how we could do that.
> > >
> > > Thanks,
> > >
> > > - Paul
> > >
> > > On Mon, Jan 3, 2022 at 12:54 PM Ted Dunning <te...@gmail.com>
> > wrote:
> > >
> > >> Christian,
> > >>
> > >> Your thoughts are very helpful. I find Arrow very nice (I use it in
> > Agstack
> > >> with Julia and Python).
> > >>
> > >> I don't think anybody is saying that Drill wouldn't be well set with a
> > >> switch to Arrow or even just interfaces to Arrow. But it is a lot of
> > work
> > >> to make it all happen.
> > >>
> > >>
> > >>
> > >> On Mon, Jan 3, 2022 at 11:37 AM Z0ltrix <z0...@pm.me.invalid>
> wrote:
> > >>
> > >>> Hi Charles, Ted, and the others here,
> > >>>
> > >>> it is very interesting to hear the evolution of Drill, Dremio and
> Arrow
> > >> in
> > >>> that context and thank you Charles for restarting that discussion.
> > >>>
> > >>> I think, and James mentioned this in the PR as well, that Drill could
> > >>> benefit from the continues progress, the Arrow project has made since
> > its
> > >>> separation from Drill. And the arrow Community seems to be large, so
> i
> > >>> assume this goes on and on with improvements, new features, etc. but
> i
> > >> have
> > >>> not enough experience in Drill internals to have an Idea in which
> mass
> > of
> > >>> refactoring this would lead.
> > >>>
> > >>> In addition to that, im not aware of the current roadmap of Arrow and
> > if
> > >>> these would fit into Drills roadmap. Maybe Arrow would go into a
> > >> different
> > >>> direction than Drill and what should we do, if Drill is bound to
> Arrow
> > >> then?
> > >>>
> > >>> On the other hand, Arrow could help Drill to a wider adoption with
> > >> clients
> > >>> like pyarrow, arrow-flight, various other programming languages etc.
> > and
> > >>> (im not sure about that) maybe its a performance benefit if Drill use
> > >> Arrow
> > >>> to read Data from HDFS(example), useses Arrow to work with it during
> > >>> execution and gives the vectors directly to my Python(example)
> programm
> > >> via
> > >>> arrow-flight so that i can Play around with Pandas, etc.
> > >>>
> > >>> Just some thoughts i have since i have used Dremio with pyarrow and
> > Drill
> > >>> with odbc connections.
> > >>>
> > >>> Regards
> > >>> Christian
> > >>> -------- Original-Nachricht --------
> > >>> Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
> > >>>
> > >>>
> > >>> Thanks Ted for the perspective! I had always wished to be a "fly on
> the
> > >>> wall" in those conversations. :-)
> > >>> -- C
> > >>>
> > >>>> On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com>
> wrote:
> > >>>>
> > >>>> Hello all,
> > >>>> There was a discussion in a recently closed PR [1] with a discussion
> > >>> between z0ltrix, James Turton and a few others about integrating
> Drill
> > >> with
> > >>> Apache Arrow and wondering why it was never done. I'd like to share
> my
> > >>> perspective as someone who has been around Drill for some time but
> also
> > >> as
> > >>> someone who never worked for MapR or Dremio. This just represents my
> > >>> understanding of events as an outsider, and I could be wrong about
> some
> > >> or
> > >>> all of this. Please forgive (or correct) any inaccuracies.
> > >>>>
> > >>>> When I first learned of Arrow and the idea of integrating Arrow with
> > >>> Drill, the thing that interested me the most was the ability to move
> > data
> > >>> between platforms without having to serialize/deserialize the data.
> > From
> > >> my
> > >>> understanding, MapR did some research and didn't find a significant
> > >>> performance advantage and hence didn't really pursue the integration.
> > The
> > >>> other side of it was that it would require a significant amount of
> work
> > >> to
> > >>> refactor major parts of Drill.
> > >>>>
> > >>>> I don't know the internal politics, but this was one of the major
> > >> points
> > >>> of diversion between Dremio and Drill.
> > >>>>
> > >>>> With that said, there was a renewed discussion on the list [2] where
> > >>> Paul Rogers proposed what he described as a "Crude but Effective"
> > >> approach
> > >>> to an Arrow integration.
> > >>>>
> > >>>> This is in the email link but here was a part of Paul's email:
> > >>>>
> > >>>>> Charles, just brainstorming a bit, I think the easiest way to start
> > is
> > >>> to create a simple, stand-alone server that speaks Arrow to the
> client,
> > >> and
> > >>> uses the native Drill client to speak to Drill. The native Drill
> client
> > >>> exposes Drill value vectors. One trick would be to convert Drill
> > vectors
> > >> to
> > >>> the Arrow format. I think that data vectors are the same format.
> > Possibly
> > >>> offset vectors. I think Arrow went its own way with null-value
> (Drill's
> > >>> is-set) vectors. So, some conversion might be a no-op, others might
> > need
> > >> to
> > >>> rewrite a vector. Good thing, this is purely at the vector level, so
> > >> would
> > >>> be easy to write. The next issue is the one that Parth has long
> pointed
> > >>> out: Drill and Arrow each have their own memory allocators. How could
> > we
> > >>> share a data vector between the two? The simplest initial solution is
> > >> just
> > >>> to copy the data from Drill to Arrow. Slow, but transparent to the
> > >> client.
> > >>> A crude first-approximation of the development steps:
> > >>>>>
> > >>>>> A crude first-approximation of the development steps:
> > >>>>> 1. Create the client shell server.
> > >>>>> 2. Implement the Arrow client protocol. Need some way to accept a
> > >> query
> > >>> and return batches of results.
> > >>>>> 3. Forward the query to Drill using the native Drill client.
> > >>>>> 4. As a first pass, copy vectors from Drill to Arrow and return
> them
> > >> to
> > >>> the client.
> > >>>>> 5. Then, solve that memory allocator problem to pass data without
> > >>> copying.
> > >>>>
> > >>>> One point that Paul made was that these pieces are fairly discrete
> and
> > >>> could be implemented without refactoring major components of Drill.
> Of
> > >>> course, this could be something for Drill 2.0. At a minimum, could we
> > >> take
> > >>> the conversation off of the PR and put it in the email list? ;-)
> > >>>>
> > >>>> Let's discuss... All ideas are welcome!
> > >>>>
> > >>>> Best,
> > >>>> -- C
> > >>>>
> > >>>>
> > >>>> [1]: https://github.com/apache/drill/pull/2412 <
> > >>> https://github.com/apache/drill/pull/2412>
> > >>>> [2]:
> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l
> > >> <
> > >>> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Paul Rogers <pa...@gmail.com>.
Hi Charles,
The material is rather dense and benefits from the Github formatting. To
preserve it, perhaps we can copy it to a subpage of the Drill 2.0 wiki page.
For now, the link to the discussion is [1]. Since the Wiki is not good for
discussions, let's have that discussion here (if anyone is up to tackling
such a weighty subject.)
Thanks,
- Paul
[1] https://github.com/apache/drill/pull/2412
On Mon, Jan 3, 2022 at 5:15 PM Charles Givre <cg...@gmail.com> wrote:
> @Paul,
> Do you mind if I copy the contents of your response to DRILL-8088 to this
> thread? There's a lot of good info there, and I'd hate to see it get lost.
> -- C
>
> > On Jan 3, 2022, at 7:41 PM, Paul Rogers <pa...@gmail.com> wrote:
> >
> > Hi All,
> >
> > Thanks Charles for dredging up that old discussion, your memory is better
> > than mine! And, thanks Ted for that summary of MapR history. As one of
> the
> > "replacement crew" brought in after the original folks left, your
> > description is consistent with my memory of events. Moreover, as we
> looked
> > at what was needed to run Drill in production, an Arrow port was far down
> > on the list: it would not have solved actual customer problems.
> >
> > Before we get too excited about Arrow, I think we should have a
> discussion
> > about what we want in an internal storage format. I added a long (sorry)
> > set of comments in that PR that Charles mentioned that tries to debunk
> the
> > myths that have grown up around using a columnar format as the internal
> > representation for a query engine. (Columnar is great for storage.) The
> > note presents the many issues we've encountered over the years that have
> > caused us to layer ever more code on top of vectors to solve various
> > problems. It also highlights a distributed-systems problem which vectors
> > make far worse.
> >
> > Arrow is meant to be portable, as Ted discussed, but it is still
> columnar,
> > and this is the source of endless problems in an execution engine. So, we
> > want to ask, what is the optimal format for what Drill actually does? I'm
> > now of the opinion that Drill might actually better benefit from a
> > row-based format, similar to what Impala uses. The notes even paint a
> path
> > forward.
> >
> > Ted's description of the goal for Demio suggests that Arrow might be the
> > right answer for that market. Drill, however, tends to be used to query
> > myriad data sources at scale and as a "query integrator" across systems.
> > This use case has different needs, which may be better served with a
> > row-based format.
> >
> > The upshot is that "value vectors vs. Arrow" is the wrong place to start
> > the discussion. The right place is "what does our many years of
> experience
> > with Drill suggest is the most efficient format for how Drill is actually
> > used?"
> >
> > Note that Drill could have an Arrow-based API independent of the internal
> > format. The quote from Charles explains how we could do that.
> >
> > Thanks,
> >
> > - Paul
> >
> > On Mon, Jan 3, 2022 at 12:54 PM Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> Christian,
> >>
> >> Your thoughts are very helpful. I find Arrow very nice (I use it in
> Agstack
> >> with Julia and Python).
> >>
> >> I don't think anybody is saying that Drill wouldn't be well set with a
> >> switch to Arrow or even just interfaces to Arrow. But it is a lot of
> work
> >> to make it all happen.
> >>
> >>
> >>
> >> On Mon, Jan 3, 2022 at 11:37 AM Z0ltrix <z0...@pm.me.invalid> wrote:
> >>
> >>> Hi Charles, Ted, and the others here,
> >>>
> >>> it is very interesting to hear the evolution of Drill, Dremio and Arrow
> >> in
> >>> that context and thank you Charles for restarting that discussion.
> >>>
> >>> I think, and James mentioned this in the PR as well, that Drill could
> >>> benefit from the continues progress, the Arrow project has made since
> its
> >>> separation from Drill. And the arrow Community seems to be large, so i
> >>> assume this goes on and on with improvements, new features, etc. but i
> >> have
> >>> not enough experience in Drill internals to have an Idea in which mass
> of
> >>> refactoring this would lead.
> >>>
> >>> In addition to that, im not aware of the current roadmap of Arrow and
> if
> >>> these would fit into Drills roadmap. Maybe Arrow would go into a
> >> different
> >>> direction than Drill and what should we do, if Drill is bound to Arrow
> >> then?
> >>>
> >>> On the other hand, Arrow could help Drill to a wider adoption with
> >> clients
> >>> like pyarrow, arrow-flight, various other programming languages etc.
> and
> >>> (im not sure about that) maybe its a performance benefit if Drill use
> >> Arrow
> >>> to read Data from HDFS(example), useses Arrow to work with it during
> >>> execution and gives the vectors directly to my Python(example) programm
> >> via
> >>> arrow-flight so that i can Play around with Pandas, etc.
> >>>
> >>> Just some thoughts i have since i have used Dremio with pyarrow and
> Drill
> >>> with odbc connections.
> >>>
> >>> Regards
> >>> Christian
> >>> -------- Original-Nachricht --------
> >>> Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
> >>>
> >>>
> >>> Thanks Ted for the perspective! I had always wished to be a "fly on the
> >>> wall" in those conversations. :-)
> >>> -- C
> >>>
> >>>> On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
> >>>>
> >>>> Hello all,
> >>>> There was a discussion in a recently closed PR [1] with a discussion
> >>> between z0ltrix, James Turton and a few others about integrating Drill
> >> with
> >>> Apache Arrow and wondering why it was never done. I'd like to share my
> >>> perspective as someone who has been around Drill for some time but also
> >> as
> >>> someone who never worked for MapR or Dremio. This just represents my
> >>> understanding of events as an outsider, and I could be wrong about some
> >> or
> >>> all of this. Please forgive (or correct) any inaccuracies.
> >>>>
> >>>> When I first learned of Arrow and the idea of integrating Arrow with
> >>> Drill, the thing that interested me the most was the ability to move
> data
> >>> between platforms without having to serialize/deserialize the data.
> From
> >> my
> >>> understanding, MapR did some research and didn't find a significant
> >>> performance advantage and hence didn't really pursue the integration.
> The
> >>> other side of it was that it would require a significant amount of work
> >> to
> >>> refactor major parts of Drill.
> >>>>
> >>>> I don't know the internal politics, but this was one of the major
> >> points
> >>> of diversion between Dremio and Drill.
> >>>>
> >>>> With that said, there was a renewed discussion on the list [2] where
> >>> Paul Rogers proposed what he described as a "Crude but Effective"
> >> approach
> >>> to an Arrow integration.
> >>>>
> >>>> This is in the email link but here was a part of Paul's email:
> >>>>
> >>>>> Charles, just brainstorming a bit, I think the easiest way to start
> is
> >>> to create a simple, stand-alone server that speaks Arrow to the client,
> >> and
> >>> uses the native Drill client to speak to Drill. The native Drill client
> >>> exposes Drill value vectors. One trick would be to convert Drill
> vectors
> >> to
> >>> the Arrow format. I think that data vectors are the same format.
> Possibly
> >>> offset vectors. I think Arrow went its own way with null-value (Drill's
> >>> is-set) vectors. So, some conversion might be a no-op, others might
> need
> >> to
> >>> rewrite a vector. Good thing, this is purely at the vector level, so
> >> would
> >>> be easy to write. The next issue is the one that Parth has long pointed
> >>> out: Drill and Arrow each have their own memory allocators. How could
> we
> >>> share a data vector between the two? The simplest initial solution is
> >> just
> >>> to copy the data from Drill to Arrow. Slow, but transparent to the
> >> client.
> >>> A crude first-approximation of the development steps:
> >>>>>
> >>>>> A crude first-approximation of the development steps:
> >>>>> 1. Create the client shell server.
> >>>>> 2. Implement the Arrow client protocol. Need some way to accept a
> >> query
> >>> and return batches of results.
> >>>>> 3. Forward the query to Drill using the native Drill client.
> >>>>> 4. As a first pass, copy vectors from Drill to Arrow and return them
> >> to
> >>> the client.
> >>>>> 5. Then, solve that memory allocator problem to pass data without
> >>> copying.
> >>>>
> >>>> One point that Paul made was that these pieces are fairly discrete and
> >>> could be implemented without refactoring major components of Drill. Of
> >>> course, this could be something for Drill 2.0. At a minimum, could we
> >> take
> >>> the conversation off of the PR and put it in the email list? ;-)
> >>>>
> >>>> Let's discuss... All ideas are welcome!
> >>>>
> >>>> Best,
> >>>> -- C
> >>>>
> >>>>
> >>>> [1]: https://github.com/apache/drill/pull/2412 <
> >>> https://github.com/apache/drill/pull/2412>
> >>>> [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l
> >> <
> >>> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Charles Givre <cg...@gmail.com>.
@Paul,
Do you mind if I copy the contents of your response to DRILL-8088 to this thread? There's a lot of good info there, and I'd hate to see it get lost.
-- C
> On Jan 3, 2022, at 7:41 PM, Paul Rogers <pa...@gmail.com> wrote:
>
> Hi All,
>
> Thanks Charles for dredging up that old discussion, your memory is better
> than mine! And, thanks Ted for that summary of MapR history. As one of the
> "replacement crew" brought in after the original folks left, your
> description is consistent with my memory of events. Moreover, as we looked
> at what was needed to run Drill in production, an Arrow port was far down
> on the list: it would not have solved actual customer problems.
>
> Before we get too excited about Arrow, I think we should have a discussion
> about what we want in an internal storage format. I added a long (sorry)
> set of comments in that PR that Charles mentioned that tries to debunk the
> myths that have grown up around using a columnar format as the internal
> representation for a query engine. (Columnar is great for storage.) The
> note presents the many issues we've encountered over the years that have
> caused us to layer ever more code on top of vectors to solve various
> problems. It also highlights a distributed-systems problem which vectors
> make far worse.
>
> Arrow is meant to be portable, as Ted discussed, but it is still columnar,
> and this is the source of endless problems in an execution engine. So, we
> want to ask, what is the optimal format for what Drill actually does? I'm
> now of the opinion that Drill might actually better benefit from a
> row-based format, similar to what Impala uses. The notes even paint a path
> forward.
>
> Ted's description of the goal for Demio suggests that Arrow might be the
> right answer for that market. Drill, however, tends to be used to query
> myriad data sources at scale and as a "query integrator" across systems.
> This use case has different needs, which may be better served with a
> row-based format.
>
> The upshot is that "value vectors vs. Arrow" is the wrong place to start
> the discussion. The right place is "what does our many years of experience
> with Drill suggest is the most efficient format for how Drill is actually
> used?"
>
> Note that Drill could have an Arrow-based API independent of the internal
> format. The quote from Charles explains how we could do that.
>
> Thanks,
>
> - Paul
>
> On Mon, Jan 3, 2022 at 12:54 PM Ted Dunning <te...@gmail.com> wrote:
>
>> Christian,
>>
>> Your thoughts are very helpful. I find Arrow very nice (I use it in Agstack
>> with Julia and Python).
>>
>> I don't think anybody is saying that Drill wouldn't be well set with a
>> switch to Arrow or even just interfaces to Arrow. But it is a lot of work
>> to make it all happen.
>>
>>
>>
>> On Mon, Jan 3, 2022 at 11:37 AM Z0ltrix <z0...@pm.me.invalid> wrote:
>>
>>> Hi Charles, Ted, and the others here,
>>>
>>> it is very interesting to hear the evolution of Drill, Dremio and Arrow
>> in
>>> that context and thank you Charles for restarting that discussion.
>>>
>>> I think, and James mentioned this in the PR as well, that Drill could
>>> benefit from the continues progress, the Arrow project has made since its
>>> separation from Drill. And the arrow Community seems to be large, so i
>>> assume this goes on and on with improvements, new features, etc. but i
>> have
>>> not enough experience in Drill internals to have an Idea in which mass of
>>> refactoring this would lead.
>>>
>>> In addition to that, im not aware of the current roadmap of Arrow and if
>>> these would fit into Drills roadmap. Maybe Arrow would go into a
>> different
>>> direction than Drill and what should we do, if Drill is bound to Arrow
>> then?
>>>
>>> On the other hand, Arrow could help Drill to a wider adoption with
>> clients
>>> like pyarrow, arrow-flight, various other programming languages etc. and
>>> (im not sure about that) maybe its a performance benefit if Drill use
>> Arrow
>>> to read Data from HDFS(example), useses Arrow to work with it during
>>> execution and gives the vectors directly to my Python(example) programm
>> via
>>> arrow-flight so that i can Play around with Pandas, etc.
>>>
>>> Just some thoughts i have since i have used Dremio with pyarrow and Drill
>>> with odbc connections.
>>>
>>> Regards
>>> Christian
>>> -------- Original-Nachricht --------
>>> Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
>>>
>>>
>>> Thanks Ted for the perspective! I had always wished to be a "fly on the
>>> wall" in those conversations. :-)
>>> -- C
>>>
>>>> On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
>>>>
>>>> Hello all,
>>>> There was a discussion in a recently closed PR [1] with a discussion
>>> between z0ltrix, James Turton and a few others about integrating Drill
>> with
>>> Apache Arrow and wondering why it was never done. I'd like to share my
>>> perspective as someone who has been around Drill for some time but also
>> as
>>> someone who never worked for MapR or Dremio. This just represents my
>>> understanding of events as an outsider, and I could be wrong about some
>> or
>>> all of this. Please forgive (or correct) any inaccuracies.
>>>>
>>>> When I first learned of Arrow and the idea of integrating Arrow with
>>> Drill, the thing that interested me the most was the ability to move data
>>> between platforms without having to serialize/deserialize the data. From
>> my
>>> understanding, MapR did some research and didn't find a significant
>>> performance advantage and hence didn't really pursue the integration. The
>>> other side of it was that it would require a significant amount of work
>> to
>>> refactor major parts of Drill.
>>>>
>>>> I don't know the internal politics, but this was one of the major
>> points
>>> of diversion between Dremio and Drill.
>>>>
>>>> With that said, there was a renewed discussion on the list [2] where
>>> Paul Rogers proposed what he described as a "Crude but Effective"
>> approach
>>> to an Arrow integration.
>>>>
>>>> This is in the email link but here was a part of Paul's email:
>>>>
>>>>> Charles, just brainstorming a bit, I think the easiest way to start is
>>> to create a simple, stand-alone server that speaks Arrow to the client,
>> and
>>> uses the native Drill client to speak to Drill. The native Drill client
>>> exposes Drill value vectors. One trick would be to convert Drill vectors
>> to
>>> the Arrow format. I think that data vectors are the same format. Possibly
>>> offset vectors. I think Arrow went its own way with null-value (Drill's
>>> is-set) vectors. So, some conversion might be a no-op, others might need
>> to
>>> rewrite a vector. Good thing, this is purely at the vector level, so
>> would
>>> be easy to write. The next issue is the one that Parth has long pointed
>>> out: Drill and Arrow each have their own memory allocators. How could we
>>> share a data vector between the two? The simplest initial solution is
>> just
>>> to copy the data from Drill to Arrow. Slow, but transparent to the
>> client.
>>> A crude first-approximation of the development steps:
>>>>>
>>>>> A crude first-approximation of the development steps:
>>>>> 1. Create the client shell server.
>>>>> 2. Implement the Arrow client protocol. Need some way to accept a
>> query
>>> and return batches of results.
>>>>> 3. Forward the query to Drill using the native Drill client.
>>>>> 4. As a first pass, copy vectors from Drill to Arrow and return them
>> to
>>> the client.
>>>>> 5. Then, solve that memory allocator problem to pass data without
>>> copying.
>>>>
>>>> One point that Paul made was that these pieces are fairly discrete and
>>> could be implemented without refactoring major components of Drill. Of
>>> course, this could be something for Drill 2.0. At a minimum, could we
>> take
>>> the conversation off of the PR and put it in the email list? ;-)
>>>>
>>>> Let's discuss... All ideas are welcome!
>>>>
>>>> Best,
>>>> -- C
>>>>
>>>>
>>>> [1]: https://github.com/apache/drill/pull/2412 <
>>> https://github.com/apache/drill/pull/2412>
>>>> [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l
>> <
>>> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
>>>>
>>>>
>>>>
>>>
>>>
>>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Paul Rogers <pa...@gmail.com>.
Hi All,
Thanks Charles for dredging up that old discussion, your memory is better
than mine! And, thanks Ted for that summary of MapR history. As one of the
"replacement crew" brought in after the original folks left, your
description is consistent with my memory of events. Moreover, as we looked
at what was needed to run Drill in production, an Arrow port was far down
on the list: it would not have solved actual customer problems.
Before we get too excited about Arrow, I think we should have a discussion
about what we want in an internal storage format. I added a long (sorry)
set of comments in that PR that Charles mentioned that tries to debunk the
myths that have grown up around using a columnar format as the internal
representation for a query engine. (Columnar is great for storage.) The
note presents the many issues we've encountered over the years that have
caused us to layer ever more code on top of vectors to solve various
problems. It also highlights a distributed-systems problem which vectors
make far worse.
Arrow is meant to be portable, as Ted discussed, but it is still columnar,
and this is the source of endless problems in an execution engine. So, we
want to ask, what is the optimal format for what Drill actually does? I'm
now of the opinion that Drill might actually better benefit from a
row-based format, similar to what Impala uses. The notes even paint a path
forward.
Ted's description of the goal for Demio suggests that Arrow might be the
right answer for that market. Drill, however, tends to be used to query
myriad data sources at scale and as a "query integrator" across systems.
This use case has different needs, which may be better served with a
row-based format.
The upshot is that "value vectors vs. Arrow" is the wrong place to start
the discussion. The right place is "what does our many years of experience
with Drill suggest is the most efficient format for how Drill is actually
used?"
Note that Drill could have an Arrow-based API independent of the internal
format. The quote from Charles explains how we could do that.
Thanks,
- Paul
On Mon, Jan 3, 2022 at 12:54 PM Ted Dunning <te...@gmail.com> wrote:
> Christian,
>
> Your thoughts are very helpful. I find Arrow very nice (I use it in Agstack
> with Julia and Python).
>
> I don't think anybody is saying that Drill wouldn't be well set with a
> switch to Arrow or even just interfaces to Arrow. But it is a lot of work
> to make it all happen.
>
>
>
> On Mon, Jan 3, 2022 at 11:37 AM Z0ltrix <z0...@pm.me.invalid> wrote:
>
> > Hi Charles, Ted, and the others here,
> >
> > it is very interesting to hear the evolution of Drill, Dremio and Arrow
> in
> > that context and thank you Charles for restarting that discussion.
> >
> > I think, and James mentioned this in the PR as well, that Drill could
> > benefit from the continues progress, the Arrow project has made since its
> > separation from Drill. And the arrow Community seems to be large, so i
> > assume this goes on and on with improvements, new features, etc. but i
> have
> > not enough experience in Drill internals to have an Idea in which mass of
> > refactoring this would lead.
> >
> > In addition to that, im not aware of the current roadmap of Arrow and if
> > these would fit into Drills roadmap. Maybe Arrow would go into a
> different
> > direction than Drill and what should we do, if Drill is bound to Arrow
> then?
> >
> > On the other hand, Arrow could help Drill to a wider adoption with
> clients
> > like pyarrow, arrow-flight, various other programming languages etc. and
> > (im not sure about that) maybe its a performance benefit if Drill use
> Arrow
> > to read Data from HDFS(example), useses Arrow to work with it during
> > execution and gives the vectors directly to my Python(example) programm
> via
> > arrow-flight so that i can Play around with Pandas, etc.
> >
> > Just some thoughts i have since i have used Dremio with pyarrow and Drill
> > with odbc connections.
> >
> > Regards
> > Christian
> > -------- Original-Nachricht --------
> > Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
> >
> >
> > Thanks Ted for the perspective! I had always wished to be a "fly on the
> > wall" in those conversations. :-)
> > -- C
> >
> > > On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
> > >
> > > Hello all,
> > > There was a discussion in a recently closed PR [1] with a discussion
> > between z0ltrix, James Turton and a few others about integrating Drill
> with
> > Apache Arrow and wondering why it was never done. I'd like to share my
> > perspective as someone who has been around Drill for some time but also
> as
> > someone who never worked for MapR or Dremio. This just represents my
> > understanding of events as an outsider, and I could be wrong about some
> or
> > all of this. Please forgive (or correct) any inaccuracies.
> > >
> > > When I first learned of Arrow and the idea of integrating Arrow with
> > Drill, the thing that interested me the most was the ability to move data
> > between platforms without having to serialize/deserialize the data. From
> my
> > understanding, MapR did some research and didn't find a significant
> > performance advantage and hence didn't really pursue the integration. The
> > other side of it was that it would require a significant amount of work
> to
> > refactor major parts of Drill.
> > >
> > > I don't know the internal politics, but this was one of the major
> points
> > of diversion between Dremio and Drill.
> > >
> > > With that said, there was a renewed discussion on the list [2] where
> > Paul Rogers proposed what he described as a "Crude but Effective"
> approach
> > to an Arrow integration.
> > >
> > > This is in the email link but here was a part of Paul's email:
> > >
> > >> Charles, just brainstorming a bit, I think the easiest way to start is
> > to create a simple, stand-alone server that speaks Arrow to the client,
> and
> > uses the native Drill client to speak to Drill. The native Drill client
> > exposes Drill value vectors. One trick would be to convert Drill vectors
> to
> > the Arrow format. I think that data vectors are the same format. Possibly
> > offset vectors. I think Arrow went its own way with null-value (Drill's
> > is-set) vectors. So, some conversion might be a no-op, others might need
> to
> > rewrite a vector. Good thing, this is purely at the vector level, so
> would
> > be easy to write. The next issue is the one that Parth has long pointed
> > out: Drill and Arrow each have their own memory allocators. How could we
> > share a data vector between the two? The simplest initial solution is
> just
> > to copy the data from Drill to Arrow. Slow, but transparent to the
> client.
> > A crude first-approximation of the development steps:
> > >>
> > >> A crude first-approximation of the development steps:
> > >> 1. Create the client shell server.
> > >> 2. Implement the Arrow client protocol. Need some way to accept a
> query
> > and return batches of results.
> > >> 3. Forward the query to Drill using the native Drill client.
> > >> 4. As a first pass, copy vectors from Drill to Arrow and return them
> to
> > the client.
> > >> 5. Then, solve that memory allocator problem to pass data without
> > copying.
> > >
> > > One point that Paul made was that these pieces are fairly discrete and
> > could be implemented without refactoring major components of Drill. Of
> > course, this could be something for Drill 2.0. At a minimum, could we
> take
> > the conversation off of the PR and put it in the email list? ;-)
> > >
> > > Let's discuss... All ideas are welcome!
> > >
> > > Best,
> > > -- C
> > >
> > >
> > > [1]: https://github.com/apache/drill/pull/2412 <
> > https://github.com/apache/drill/pull/2412>
> > > [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l
> <
> > https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
> > >
> > >
> > >
> >
> >
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Ted Dunning <te...@gmail.com>.
Christian,
Your thoughts are very helpful. I find Arrow very nice (I use it in Agstack
with Julia and Python).
I don't think anybody is saying that Drill wouldn't be well set with a
switch to Arrow or even just interfaces to Arrow. But it is a lot of work
to make it all happen.
On Mon, Jan 3, 2022 at 11:37 AM Z0ltrix <z0...@pm.me.invalid> wrote:
> Hi Charles, Ted, and the others here,
>
> it is very interesting to hear the evolution of Drill, Dremio and Arrow in
> that context and thank you Charles for restarting that discussion.
>
> I think, and James mentioned this in the PR as well, that Drill could
> benefit from the continues progress, the Arrow project has made since its
> separation from Drill. And the arrow Community seems to be large, so i
> assume this goes on and on with improvements, new features, etc. but i have
> not enough experience in Drill internals to have an Idea in which mass of
> refactoring this would lead.
>
> In addition to that, im not aware of the current roadmap of Arrow and if
> these would fit into Drills roadmap. Maybe Arrow would go into a different
> direction than Drill and what should we do, if Drill is bound to Arrow then?
>
> On the other hand, Arrow could help Drill to a wider adoption with clients
> like pyarrow, arrow-flight, various other programming languages etc. and
> (im not sure about that) maybe its a performance benefit if Drill use Arrow
> to read Data from HDFS(example), useses Arrow to work with it during
> execution and gives the vectors directly to my Python(example) programm via
> arrow-flight so that i can Play around with Pandas, etc.
>
> Just some thoughts i have since i have used Dremio with pyarrow and Drill
> with odbc connections.
>
> Regards
> Christian
> -------- Original-Nachricht --------
> Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
>
>
> Thanks Ted for the perspective! I had always wished to be a "fly on the
> wall" in those conversations. :-)
> -- C
>
> > On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
> >
> > Hello all,
> > There was a discussion in a recently closed PR [1] with a discussion
> between z0ltrix, James Turton and a few others about integrating Drill with
> Apache Arrow and wondering why it was never done. I'd like to share my
> perspective as someone who has been around Drill for some time but also as
> someone who never worked for MapR or Dremio. This just represents my
> understanding of events as an outsider, and I could be wrong about some or
> all of this. Please forgive (or correct) any inaccuracies.
> >
> > When I first learned of Arrow and the idea of integrating Arrow with
> Drill, the thing that interested me the most was the ability to move data
> between platforms without having to serialize/deserialize the data. From my
> understanding, MapR did some research and didn't find a significant
> performance advantage and hence didn't really pursue the integration. The
> other side of it was that it would require a significant amount of work to
> refactor major parts of Drill.
> >
> > I don't know the internal politics, but this was one of the major points
> of diversion between Dremio and Drill.
> >
> > With that said, there was a renewed discussion on the list [2] where
> Paul Rogers proposed what he described as a "Crude but Effective" approach
> to an Arrow integration.
> >
> > This is in the email link but here was a part of Paul's email:
> >
> >> Charles, just brainstorming a bit, I think the easiest way to start is
> to create a simple, stand-alone server that speaks Arrow to the client, and
> uses the native Drill client to speak to Drill. The native Drill client
> exposes Drill value vectors. One trick would be to convert Drill vectors to
> the Arrow format. I think that data vectors are the same format. Possibly
> offset vectors. I think Arrow went its own way with null-value (Drill's
> is-set) vectors. So, some conversion might be a no-op, others might need to
> rewrite a vector. Good thing, this is purely at the vector level, so would
> be easy to write. The next issue is the one that Parth has long pointed
> out: Drill and Arrow each have their own memory allocators. How could we
> share a data vector between the two? The simplest initial solution is just
> to copy the data from Drill to Arrow. Slow, but transparent to the client.
> A crude first-approximation of the development steps:
> >>
> >> A crude first-approximation of the development steps:
> >> 1. Create the client shell server.
> >> 2. Implement the Arrow client protocol. Need some way to accept a query
> and return batches of results.
> >> 3. Forward the query to Drill using the native Drill client.
> >> 4. As a first pass, copy vectors from Drill to Arrow and return them to
> the client.
> >> 5. Then, solve that memory allocator problem to pass data without
> copying.
> >
> > One point that Paul made was that these pieces are fairly discrete and
> could be implemented without refactoring major components of Drill. Of
> course, this could be something for Drill 2.0. At a minimum, could we take
> the conversation off of the PR and put it in the email list? ;-)
> >
> > Let's discuss... All ideas are welcome!
> >
> > Best,
> > -- C
> >
> >
> > [1]: https://github.com/apache/drill/pull/2412 <
> https://github.com/apache/drill/pull/2412>
> > [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l <
> https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
> >
> >
> >
>
>
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Z0ltrix <z0...@pm.me.INVALID>.
Hi Charles, Ted, and the others here,
it is very interesting to hear the evolution of Drill, Dremio and Arrow in that context and thank you Charles for restarting that discussion.
I think, and James mentioned this in the PR as well, that Drill could benefit from the continues progress, the Arrow project has made since its separation from Drill. And the arrow Community seems to be large, so i assume this goes on and on with improvements, new features, etc. but i have not enough experience in Drill internals to have an Idea in which mass of refactoring this would lead.
In addition to that, im not aware of the current roadmap of Arrow and if these would fit into Drills roadmap. Maybe Arrow would go into a different direction than Drill and what should we do, if Drill is bound to Arrow then?
On the other hand, Arrow could help Drill to a wider adoption with clients like pyarrow, arrow-flight, various other programming languages etc. and (im not sure about that) maybe its a performance benefit if Drill use Arrow to read Data from HDFS(example), useses Arrow to work with it during execution and gives the vectors directly to my Python(example) programm via arrow-flight so that i can Play around with Pandas, etc.
Just some thoughts i have since i have used Dremio with pyarrow and Drill with odbc connections.
Regards
Christian
\-------- Original-Nachricht --------
Am 3. Jan. 2022, 20:08, Charles Givre schrieb:
>
>
>
> Thanks Ted for the perspective! I had always wished to be a "fly on the wall" in those conversations. :-)
> \-- C
>
> > On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
> >
> > Hello all,
> > There was a discussion in a recently closed PR \[1\] with a discussion between z0ltrix, James Turton and a few others about integrating Drill with Apache Arrow and wondering why it was never done. I'd like to share my perspective as someone who has been around Drill for some time but also as someone who never worked for MapR or Dremio. This just represents my understanding of events as an outsider, and I could be wrong about some or all of this. Please forgive (or correct) any inaccuracies.
> >
> > When I first learned of Arrow and the idea of integrating Arrow with Drill, the thing that interested me the most was the ability to move data between platforms without having to serialize/deserialize the data. From my understanding, MapR did some research and didn't find a significant performance advantage and hence didn't really pursue the integration. The other side of it was that it would require a significant amount of work to refactor major parts of Drill.
> >
> > I don't know the internal politics, but this was one of the major points of diversion between Dremio and Drill.
> >
> > With that said, there was a renewed discussion on the list \[2\] where Paul Rogers proposed what he described as a "Crude but Effective" approach to an Arrow integration.
> >
> > This is in the email link but here was a part of Paul's email:
> >
> >> Charles, just brainstorming a bit, I think the easiest way to start is to create a simple, stand-alone server that speaks Arrow to the client, and uses the native Drill client to speak to Drill. The native Drill client exposes Drill value vectors. One trick would be to convert Drill vectors to the Arrow format. I think that data vectors are the same format. Possibly offset vectors. I think Arrow went its own way with null-value (Drill's is-set) vectors. So, some conversion might be a no-op, others might need to rewrite a vector. Good thing, this is purely at the vector level, so would be easy to write. The next issue is the one that Parth has long pointed out: Drill and Arrow each have their own memory allocators. How could we share a data vector between the two? The simplest initial solution is just to copy the data from Drill to Arrow. Slow, but transparent to the client. A crude first-approximation of the development steps:
> >>
> >> A crude first-approximation of the development steps:
> >> 1. Create the client shell server.
> >> 2. Implement the Arrow client protocol. Need some way to accept a query and return batches of results.
> >> 3. Forward the query to Drill using the native Drill client.
> >> 4. As a first pass, copy vectors from Drill to Arrow and return them to the client.
> >> 5. Then, solve that memory allocator problem to pass data without copying.
> >
> > One point that Paul made was that these pieces are fairly discrete and could be implemented without refactoring major components of Drill. Of course, this could be something for Drill 2.0. At a minimum, could we take the conversation off of the PR and put it in the email list? ;-)
> >
> > Let's discuss... All ideas are welcome!
> >
> > Best,
> > -- C
> >
> >
> > \[1\]: https://github.com/apache/drill/pull/2412 <https://github.com/apache/drill/pull/2412>
> > \[2\]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l <https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
> >
> >
> >
Re: [DISCUSS] Restarting the Arrow Conversation
Posted by Charles Givre <cg...@gmail.com>.
Thanks Ted for the perspective! I had always wished to be a "fly on the wall" in those conversations. :-)
-- C
> On Jan 3, 2022, at 11:00 AM, Charles Givre <cg...@gmail.com> wrote:
>
> Hello all,
> There was a discussion in a recently closed PR [1] with a discussion between z0ltrix, James Turton and a few others about integrating Drill with Apache Arrow and wondering why it was never done. I'd like to share my perspective as someone who has been around Drill for some time but also as someone who never worked for MapR or Dremio. This just represents my understanding of events as an outsider, and I could be wrong about some or all of this. Please forgive (or correct) any inaccuracies.
>
> When I first learned of Arrow and the idea of integrating Arrow with Drill, the thing that interested me the most was the ability to move data between platforms without having to serialize/deserialize the data. From my understanding, MapR did some research and didn't find a significant performance advantage and hence didn't really pursue the integration. The other side of it was that it would require a significant amount of work to refactor major parts of Drill.
>
> I don't know the internal politics, but this was one of the major points of diversion between Dremio and Drill.
>
> With that said, there was a renewed discussion on the list [2] where Paul Rogers proposed what he described as a "Crude but Effective" approach to an Arrow integration.
>
> This is in the email link but here was a part of Paul's email:
>
>> Charles, just brainstorming a bit, I think the easiest way to start is to create a simple, stand-alone server that speaks Arrow to the client, and uses the native Drill client to speak to Drill. The native Drill client exposes Drill value vectors. One trick would be to convert Drill vectors to the Arrow format. I think that data vectors are the same format. Possibly offset vectors. I think Arrow went its own way with null-value (Drill's is-set) vectors. So, some conversion might be a no-op, others might need to rewrite a vector. Good thing, this is purely at the vector level, so would be easy to write. The next issue is the one that Parth has long pointed out: Drill and Arrow each have their own memory allocators. How could we share a data vector between the two? The simplest initial solution is just to copy the data from Drill to Arrow. Slow, but transparent to the client. A crude first-approximation of the development steps:
>>
>> A crude first-approximation of the development steps:
>> 1. Create the client shell server.
>> 2. Implement the Arrow client protocol. Need some way to accept a query and return batches of results.
>> 3. Forward the query to Drill using the native Drill client.
>> 4. As a first pass, copy vectors from Drill to Arrow and return them to the client.
>> 5. Then, solve that memory allocator problem to pass data without copying.
>
> One point that Paul made was that these pieces are fairly discrete and could be implemented without refactoring major components of Drill. Of course, this could be something for Drill 2.0. At a minimum, could we take the conversation off of the PR and put it in the email list? ;-)
>
> Let's discuss... All ideas are welcome!
>
> Best,
> -- C
>
>
> [1]: https://github.com/apache/drill/pull/2412 <https://github.com/apache/drill/pull/2412>
> [2]: https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l <https://lists.apache.org/thread/hcmygrv8q8jyw8p57fm9qy3vw2kqfr5l>
>
>
>