You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andy Grove <an...@gmail.com> on 2019/07/20 17:17:55 UTC
[DISCUSS] [Gandiva] Adding query plan to Gandiva protobuf definition
I recently created a small PoC of distributed query execution on Kubernetes
using the Rust implementation of Apache Arrow and the DataFusion query
engine [1].
This PoC uses gRPC to pass query plans to executor nodes and the proto file
[2] is largely based on the Gandiva proto file [3]. The PoC is very basic
but I think it demonstrates the power of having query plans as part of the
proto file. This would allow distributed applications to be built based on
Arrow standards in a way that is not dependent on any particular
implementation of Arrow and would even allow mixing and matching query
engines.
I wanted to start this discussion to see what the appetite is here for
accepting PRs to add query plan structures to the Gandiva proto file and
also whether we can consider making this an Arrow proto file rather than
being Gandiva-specific, over time.
Thanks,
Andy.
[1] https://github.com/andygrove/ballista
[2]
https://github.com/andygrove/ballista/blob/master/proto/ballista/ballista.proto
[3]
https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
Re: [DISCUSS] [Gandiva] Adding query plan to Gandiva protobuf definition
Posted by Andy Grove <an...@gmail.com>.
The outcome of this discussion is that I wasn't able to really make a good
case for this, so am abandoning this effort. There aren't enough compelling
reasons to do this and it will just cause confusion.
I appreciate having the opportunity to have the discussion.
Thanks,
Andy.
On Mon, Jul 22, 2019 at 11:50 AM Andy Grove <an...@gmail.com> wrote:
> Thanks, Jacques and Wes.
>
> I agree that this needs discussion and a design document. I have put
> together this Google doc to get the ball rolling:
>
>
> https://docs.google.com/document/d/1Uv1FmPs7uYMLoJUH1EF0oxm-ujtz1h1tJFl0zN60TIg/edit?usp=sharing
>
> Thanks,
>
> Andy.
>
> On Mon, Jul 22, 2019 at 6:39 AM Wes McKinney <we...@gmail.com> wrote:
>
>> I agree that I'd also like to see a design / goals document so clarify
>> the scope (and the non-goals, too)
>>
>> In general, I would hesitate to add anything higher level to the
>> Gandiva protos -- there is already confusion from people who believe
>> that Gandiva is a "query engine" where it is actually a query engine
>> subsystem (execution kernel compiler/generator). See for example the
>> thread just a week ago [1]
>>
>> If you add higher level query plan structures to the proto file, I
>> fear it will generate more confusion. If the plan ends up being to
>> have a larger proto file, it would be good to move it someplace that
>> isn't Gandiva-specific and clearly indicate that Gandiva is
>> responsible for code generation for certain structures in the proto.
>> We can also address some of these issues through better project
>> documentation and READMEs.
>>
>> [1]:
>> https://lists.apache.org/thread.html/212db05e98549f5938f3af41dade51d7a3e47255178a6c76652adc79@%3Cdev.arrow.apache.org%3E
>>
>> On Sun, Jul 21, 2019 at 4:23 PM Jacques Nadeau <ja...@apache.org>
>> wrote:
>> >
>> > Some thoughts:
>> >
>> > 1. I think it would make sense to start with a design
>> > discussion/document about the goals and what we think is
>> implementation
>> > specific versus generally applicable. In general, a distributed
>> execution
>> > plan seems pretty implementation specific. My sense is that you'd
>> never run
>> > a distributed execution plan outside of the knowledge of the
>> particular
>> > execution environment it is running within. Part of that is usually
>> > distributed execution also includes lifecycle management. For
>> example, if
>> > you're going to have work-stealing or early termination in your
>> execution
>> > engine, those are operations that stitch into execution coordination
>> (and
>> > thus a specific impl). If distributed execution is always engine
>> specific,
>> > why try to create a general one for multiple engines?
>> > 2. With regards to making Gandiva protos more generic: I'd like to
>> see
>> > more clarity on #1. On one hand, extending things so they are reused
>> is
>> > good. On the other hand, the more consumers of an interface, the more
>> > overloads/non-impls you have for each consumer of it.
>> >
>> >
>> > On Sat, Jul 20, 2019 at 10:18 AM Andy Grove <an...@gmail.com>
>> wrote:
>> >
>> > > I recently created a small PoC of distributed query execution on
>> Kubernetes
>> > > using the Rust implementation of Apache Arrow and the DataFusion query
>> > > engine [1].
>> > >
>> > > This PoC uses gRPC to pass query plans to executor nodes and the
>> proto file
>> > > [2] is largely based on the Gandiva proto file [3]. The PoC is very
>> basic
>> > > but I think it demonstrates the power of having query plans as part
>> of the
>> > > proto file. This would allow distributed applications to be built
>> based on
>> > > Arrow standards in a way that is not dependent on any particular
>> > > implementation of Arrow and would even allow mixing and matching query
>> > > engines.
>> > >
>> > > I wanted to start this discussion to see what the appetite is here for
>> > > accepting PRs to add query plan structures to the Gandiva proto file
>> and
>> > > also whether we can consider making this an Arrow proto file rather
>> than
>> > > being Gandiva-specific, over time.
>> > >
>> > > Thanks,
>> > >
>> > > Andy.
>> > >
>> > > [1] https://github.com/andygrove/ballista
>> > >
>> > > [2]
>> > >
>> > >
>> https://github.com/andygrove/ballista/blob/master/proto/ballista/ballista.proto
>> > >
>> > > [3]
>> > >
>> > >
>> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>> > >
>>
>
Re: [DISCUSS] [Gandiva] Adding query plan to Gandiva protobuf definition
Posted by Andy Grove <an...@gmail.com>.
Thanks, Jacques and Wes.
I agree that this needs discussion and a design document. I have put
together this Google doc to get the ball rolling:
https://docs.google.com/document/d/1Uv1FmPs7uYMLoJUH1EF0oxm-ujtz1h1tJFl0zN60TIg/edit?usp=sharing
Thanks,
Andy.
On Mon, Jul 22, 2019 at 6:39 AM Wes McKinney <we...@gmail.com> wrote:
> I agree that I'd also like to see a design / goals document so clarify
> the scope (and the non-goals, too)
>
> In general, I would hesitate to add anything higher level to the
> Gandiva protos -- there is already confusion from people who believe
> that Gandiva is a "query engine" where it is actually a query engine
> subsystem (execution kernel compiler/generator). See for example the
> thread just a week ago [1]
>
> If you add higher level query plan structures to the proto file, I
> fear it will generate more confusion. If the plan ends up being to
> have a larger proto file, it would be good to move it someplace that
> isn't Gandiva-specific and clearly indicate that Gandiva is
> responsible for code generation for certain structures in the proto.
> We can also address some of these issues through better project
> documentation and READMEs.
>
> [1]:
> https://lists.apache.org/thread.html/212db05e98549f5938f3af41dade51d7a3e47255178a6c76652adc79@%3Cdev.arrow.apache.org%3E
>
> On Sun, Jul 21, 2019 at 4:23 PM Jacques Nadeau <ja...@apache.org> wrote:
> >
> > Some thoughts:
> >
> > 1. I think it would make sense to start with a design
> > discussion/document about the goals and what we think is
> implementation
> > specific versus generally applicable. In general, a distributed
> execution
> > plan seems pretty implementation specific. My sense is that you'd
> never run
> > a distributed execution plan outside of the knowledge of the
> particular
> > execution environment it is running within. Part of that is usually
> > distributed execution also includes lifecycle management. For
> example, if
> > you're going to have work-stealing or early termination in your
> execution
> > engine, those are operations that stitch into execution coordination
> (and
> > thus a specific impl). If distributed execution is always engine
> specific,
> > why try to create a general one for multiple engines?
> > 2. With regards to making Gandiva protos more generic: I'd like to see
> > more clarity on #1. On one hand, extending things so they are reused
> is
> > good. On the other hand, the more consumers of an interface, the more
> > overloads/non-impls you have for each consumer of it.
> >
> >
> > On Sat, Jul 20, 2019 at 10:18 AM Andy Grove <an...@gmail.com>
> wrote:
> >
> > > I recently created a small PoC of distributed query execution on
> Kubernetes
> > > using the Rust implementation of Apache Arrow and the DataFusion query
> > > engine [1].
> > >
> > > This PoC uses gRPC to pass query plans to executor nodes and the proto
> file
> > > [2] is largely based on the Gandiva proto file [3]. The PoC is very
> basic
> > > but I think it demonstrates the power of having query plans as part of
> the
> > > proto file. This would allow distributed applications to be built
> based on
> > > Arrow standards in a way that is not dependent on any particular
> > > implementation of Arrow and would even allow mixing and matching query
> > > engines.
> > >
> > > I wanted to start this discussion to see what the appetite is here for
> > > accepting PRs to add query plan structures to the Gandiva proto file
> and
> > > also whether we can consider making this an Arrow proto file rather
> than
> > > being Gandiva-specific, over time.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > > [1] https://github.com/andygrove/ballista
> > >
> > > [2]
> > >
> > >
> https://github.com/andygrove/ballista/blob/master/proto/ballista/ballista.proto
> > >
> > > [3]
> > >
> > >
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> > >
>
Re: [DISCUSS] [Gandiva] Adding query plan to Gandiva protobuf definition
Posted by Wes McKinney <we...@gmail.com>.
I agree that I'd also like to see a design / goals document so clarify
the scope (and the non-goals, too)
In general, I would hesitate to add anything higher level to the
Gandiva protos -- there is already confusion from people who believe
that Gandiva is a "query engine" where it is actually a query engine
subsystem (execution kernel compiler/generator). See for example the
thread just a week ago [1]
If you add higher level query plan structures to the proto file, I
fear it will generate more confusion. If the plan ends up being to
have a larger proto file, it would be good to move it someplace that
isn't Gandiva-specific and clearly indicate that Gandiva is
responsible for code generation for certain structures in the proto.
We can also address some of these issues through better project
documentation and READMEs.
[1]: https://lists.apache.org/thread.html/212db05e98549f5938f3af41dade51d7a3e47255178a6c76652adc79@%3Cdev.arrow.apache.org%3E
On Sun, Jul 21, 2019 at 4:23 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> Some thoughts:
>
> 1. I think it would make sense to start with a design
> discussion/document about the goals and what we think is implementation
> specific versus generally applicable. In general, a distributed execution
> plan seems pretty implementation specific. My sense is that you'd never run
> a distributed execution plan outside of the knowledge of the particular
> execution environment it is running within. Part of that is usually
> distributed execution also includes lifecycle management. For example, if
> you're going to have work-stealing or early termination in your execution
> engine, those are operations that stitch into execution coordination (and
> thus a specific impl). If distributed execution is always engine specific,
> why try to create a general one for multiple engines?
> 2. With regards to making Gandiva protos more generic: I'd like to see
> more clarity on #1. On one hand, extending things so they are reused is
> good. On the other hand, the more consumers of an interface, the more
> overloads/non-impls you have for each consumer of it.
>
>
> On Sat, Jul 20, 2019 at 10:18 AM Andy Grove <an...@gmail.com> wrote:
>
> > I recently created a small PoC of distributed query execution on Kubernetes
> > using the Rust implementation of Apache Arrow and the DataFusion query
> > engine [1].
> >
> > This PoC uses gRPC to pass query plans to executor nodes and the proto file
> > [2] is largely based on the Gandiva proto file [3]. The PoC is very basic
> > but I think it demonstrates the power of having query plans as part of the
> > proto file. This would allow distributed applications to be built based on
> > Arrow standards in a way that is not dependent on any particular
> > implementation of Arrow and would even allow mixing and matching query
> > engines.
> >
> > I wanted to start this discussion to see what the appetite is here for
> > accepting PRs to add query plan structures to the Gandiva proto file and
> > also whether we can consider making this an Arrow proto file rather than
> > being Gandiva-specific, over time.
> >
> > Thanks,
> >
> > Andy.
> >
> > [1] https://github.com/andygrove/ballista
> >
> > [2]
> >
> > https://github.com/andygrove/ballista/blob/master/proto/ballista/ballista.proto
> >
> > [3]
> >
> > https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> >
Re: [DISCUSS] [Gandiva] Adding query plan to Gandiva protobuf definition
Posted by Jacques Nadeau <ja...@apache.org>.
Some thoughts:
1. I think it would make sense to start with a design
discussion/document about the goals and what we think is implementation
specific versus generally applicable. In general, a distributed execution
plan seems pretty implementation specific. My sense is that you'd never run
a distributed execution plan outside of the knowledge of the particular
execution environment it is running within. Part of that is usually
distributed execution also includes lifecycle management. For example, if
you're going to have work-stealing or early termination in your execution
engine, those are operations that stitch into execution coordination (and
thus a specific impl). If distributed execution is always engine specific,
why try to create a general one for multiple engines?
2. With regards to making Gandiva protos more generic: I'd like to see
more clarity on #1. On one hand, extending things so they are reused is
good. On the other hand, the more consumers of an interface, the more
overloads/non-impls you have for each consumer of it.
On Sat, Jul 20, 2019 at 10:18 AM Andy Grove <an...@gmail.com> wrote:
> I recently created a small PoC of distributed query execution on Kubernetes
> using the Rust implementation of Apache Arrow and the DataFusion query
> engine [1].
>
> This PoC uses gRPC to pass query plans to executor nodes and the proto file
> [2] is largely based on the Gandiva proto file [3]. The PoC is very basic
> but I think it demonstrates the power of having query plans as part of the
> proto file. This would allow distributed applications to be built based on
> Arrow standards in a way that is not dependent on any particular
> implementation of Arrow and would even allow mixing and matching query
> engines.
>
> I wanted to start this discussion to see what the appetite is here for
> accepting PRs to add query plan structures to the Gandiva proto file and
> also whether we can consider making this an Arrow proto file rather than
> being Gandiva-specific, over time.
>
> Thanks,
>
> Andy.
>
> [1] https://github.com/andygrove/ballista
>
> [2]
>
> https://github.com/andygrove/ballista/blob/master/proto/ballista/ballista.proto
>
> [3]
>
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>