You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by paddy horan <pa...@hotmail.com> on 2021/03/18 13:39:54 UTC
[DISCUSS] How to describe computation on Arrow data?
Hi All,
I do not have a computer science background so I may not be asking this in the correct way or using the correct terminology but I wonder if we can achieve some level of standardization when describing computation over Arrow data.
At the moment on the Rust side DataFusion clearly has a way to describe computation, I believe that Ballista adds the ability to serialize this to allow distributed computation. On the C++ side work is starting on a similar query engine and we already have Gandiva. Is there an opportunity to define a kind of IR for computation over Arrow data that could be adopted across implementations?
In this case DataFusion could easily incorporate Gandiva to generate optimized compute kernels if they were using the same IR to describe computation. Applications built on Arrow could "describe" computation in any language and take advantage or innovations across the community, adding this to Arrow's zero copy data sharing could be a game changer in my mind. I'm not someone who knows enough to drive this forward but I obviously would like to get involved. For some time I was playing around with using TVM's relay IR [1] and applying it to Arrow data.
As the Arrow memory format has now matured I fell like this could be the next step. Is there any plan for this kind of work or are we going to allow sub-projects to "go their own way"?
Thanks,
Paddy
[1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (apache.org)<https://tvm.apache.org/docs/dev/relay_intro.html>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Andy Grove <an...@gmail.com>.
There was a Google design doc from back in 2019 [1] where we discussed this
(or something similar at least).
I also remember reading about Weld's IR. It would be good to learn from
their work.
[1]
https://docs.google.com/document/d/1Uv1FmPs7uYMLoJUH1EF0oxm-ujtz1h1tJFl0zN60TIg/edit?usp=sharing
On Thu, Mar 18, 2021 at 9:49 AM Micah Kornfield <em...@gmail.com>
wrote:
> I think there might be discussion on two levels of computation, physical
> query execution plans, and potentially something "lower level"? When this
> has come up in the past, I was a little skeptical of constraining every SDK
> to use the same description, so I agree with Wes's point about keeping any
> spec open in the short term. Ballista as an opt-in model, does sound like
> possibly the right approach.
>
> I might be misunderstanding, but I think Weld [1] is another project
> targeting the lower level components?
>
> Also, I think there was a little bit of effort to come up with a common
> expression representation within C++, but got stalled on whether to use the
> Gandiva expression descriptions or Flatbuffers, I can't seem to find the
> thread/JIRA/discussion on this. I'll try to look some more this evening.
>
> [1] https://github.com/weld-project/weld
>
> On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
>
>> I'm interested in providing some path to make this extensible. To pick an
>> example, suppose the user wants to compute the first k principle
>> components. We've talked [1] about the possibility of incorporating richer
>> communication semantics in Ballista (a la MPI sub-communicators) and
>> numerical algorithms such as PCA would benefit. Those specific algorithms
>> wouldn't belong in Arrow or Ballista core, but I think there's an
>> opportunity for plugins to offer this sort of capability and it would be
>> lovely if the language-independent protocol could call them. Do you see a
>> good way to do this via ballista.proto?
>>
>> [1] https://github.com/ballista-compute/ballista/issues/303
>>
>> Andy Grove <an...@gmail.com> writes:
>>
>> > Hi Paddy,
>> >
>> > Thanks for raising this.
>> >
>> > Ballista defines computations using protobuf [1] to describe logical and
>> > physical query plans, which consist of operators and expressions. It is
>> > actually based on the Gandiva protobuf [2] for describing expressions.
>> >
>> > I see a lot of value in standardizing some of this across
>> implementations.
>> > Ballista is essentially becoming a distributed scheduler for Arrow and
>> can
>> > work with any implementation that supports this protobuf definition of
>> > query plans.
>> >
>> > It would also make it easier to embed C++ in Rust, or Rust in C++,
>> having
>> > this common IR, so I would be all for having something like this as an
>> > Arrow specification.
>> >
>> > Thanks,
>> >
>> > Andy.
>> >
>> > [1]
>> >
>> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
>> > [2]
>> >
>> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>> >
>> >
>> > On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
>> wrote:
>> >
>> >> Hi All,
>> >>
>> >> I do not have a computer science background so I may not be asking
>> this in
>> >> the correct way or using the correct terminology but I wonder if we can
>> >> achieve some level of standardization when describing computation over
>> >> Arrow data.
>> >>
>> >> At the moment on the Rust side DataFusion clearly has a way to describe
>> >> computation, I believe that Ballista adds the ability to serialize
>> this to
>> >> allow distributed computation. On the C++ side work is starting on a
>> >> similar query engine and we already have Gandiva. Is there an
>> opportunity
>> >> to define a kind of IR for computation over Arrow data that could be
>> >> adopted across implementations?
>> >>
>> >> In this case DataFusion could easily incorporate Gandiva to generate
>> >> optimized compute kernels if they were using the same IR to describe
>> >> computation. Applications built on Arrow could "describe" computation
>> in
>> >> any language and take advantage or innovations across the community,
>> adding
>> >> this to Arrow's zero copy data sharing could be a game changer in my
>> mind.
>> >> I'm not someone who knows enough to drive this forward but I obviously
>> >> would like to get involved. For some time I was playing around with
>> using
>> >> TVM's relay IR [1] and applying it to Arrow data.
>> >>
>> >> As the Arrow memory format has now matured I fell like this could be
>> the
>> >> next step. Is there any plan for this kind of work or are we going to
>> >> allow sub-projects to "go their own way"?
>> >>
>> >> Thanks,
>> >> Paddy
>> >>
>> >> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
>> apache.org)<
>> >> https://tvm.apache.org/docs/dev/relay_intro.html>
>> >>
>> >>
>>
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Hi,
The main benefit I see for a standard for queries would not be on a
serialization format, but on its semantics.
IMO one of the main reasons for a lack of a standard of queries at the
protobuf level is that human-readability vastly outweighs serialization -
queries are at very most a megabyte in size, and SQL (or json or toml) is
often enough, and much easier to read, share, learn, etc.
With that said, imo a core challenge of SQL today is that every engine
implements small semantic variations, which often derive from a myriad of
tradeoffs that the different engines have to make, often consequent of how
each of them represents data in-memory.
Because arrow imposes a fixed in-memory format, those tradeoffs are more
likely to be similar across implementations, and thus semantic parity is
easier to spec. (The counter argument is that different engines have
different goals / use-cases, and these use-cases induce different querying
patterns and thus different tradeoffs)
The use-case I see here would be to have an alignment on a SQL dialect and
require implementations to return the same *result* over a given IPC file;
I.e.
SQL statement + IPC files = IPC file
would hold true for every implementation.
Best,
Jorge
On Thu, Mar 18, 2021 at 5:22 PM Andrew Lamb <al...@influxdata.com> wrote:
> Any higher level physical execution plan most likely needs a way to
> represent expressions. Thus focusing initially on a standard for
> expressions might be a good way to add value but keep the scope of the
> effort reasonable
>
> On Thu, Mar 18, 2021 at 11:49 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > I think there might be discussion on two levels of computation, physical
> > query execution plans, and potentially something "lower level"? When
> this
> > has come up in the past, I was a little skeptical of constraining every
> SDK
> > to use the same description, so I agree with Wes's point about keeping
> any
> > spec open in the short term. Ballista as an opt-in model, does sound
> like
> > possibly the right approach.
> >
> > I might be misunderstanding, but I think Weld [1] is another project
> > targeting the lower level components?
> >
> > Also, I think there was a little bit of effort to come up with a common
> > expression representation within C++, but got stalled on whether to use
> the
> > Gandiva expression descriptions or Flatbuffers, I can't seem to find the
> > thread/JIRA/discussion on this. I'll try to look some more this evening.
> >
> > [1] https://github.com/weld-project/weld
> >
> > On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
> >
> > > I'm interested in providing some path to make this extensible. To pick
> an
> > > example, suppose the user wants to compute the first k principle
> > > components. We've talked [1] about the possibility of incorporating
> > richer
> > > communication semantics in Ballista (a la MPI sub-communicators) and
> > > numerical algorithms such as PCA would benefit. Those specific
> algorithms
> > > wouldn't belong in Arrow or Ballista core, but I think there's an
> > > opportunity for plugins to offer this sort of capability and it would
> be
> > > lovely if the language-independent protocol could call them. Do you
> see a
> > > good way to do this via ballista.proto?
> > >
> > > [1] https://github.com/ballista-compute/ballista/issues/303
> > >
> > > Andy Grove <an...@gmail.com> writes:
> > >
> > > > Hi Paddy,
> > > >
> > > > Thanks for raising this.
> > > >
> > > > Ballista defines computations using protobuf [1] to describe logical
> > and
> > > > physical query plans, which consist of operators and expressions. It
> is
> > > > actually based on the Gandiva protobuf [2] for describing
> expressions.
> > > >
> > > > I see a lot of value in standardizing some of this across
> > > implementations.
> > > > Ballista is essentially becoming a distributed scheduler for Arrow
> and
> > > can
> > > > work with any implementation that supports this protobuf definition
> of
> > > > query plans.
> > > >
> > > > It would also make it easier to embed C++ in Rust, or Rust in C++,
> > having
> > > > this common IR, so I would be all for having something like this as
> an
> > > > Arrow specification.
> > > >
> > > > Thanks,
> > > >
> > > > Andy.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> > > > [2]
> > > >
> > >
> >
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> > > >
> > > >
> > > > On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> > > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I do not have a computer science background so I may not be asking
> > this
> > > in
> > > >> the correct way or using the correct terminology but I wonder if we
> > can
> > > >> achieve some level of standardization when describing computation
> over
> > > >> Arrow data.
> > > >>
> > > >> At the moment on the Rust side DataFusion clearly has a way to
> > describe
> > > >> computation, I believe that Ballista adds the ability to serialize
> > this
> > > to
> > > >> allow distributed computation. On the C++ side work is starting on
> a
> > > >> similar query engine and we already have Gandiva. Is there an
> > > opportunity
> > > >> to define a kind of IR for computation over Arrow data that could be
> > > >> adopted across implementations?
> > > >>
> > > >> In this case DataFusion could easily incorporate Gandiva to generate
> > > >> optimized compute kernels if they were using the same IR to describe
> > > >> computation. Applications built on Arrow could "describe"
> computation
> > > in
> > > >> any language and take advantage or innovations across the community,
> > > adding
> > > >> this to Arrow's zero copy data sharing could be a game changer in my
> > > mind.
> > > >> I'm not someone who knows enough to drive this forward but I
> obviously
> > > >> would like to get involved. For some time I was playing around with
> > > using
> > > >> TVM's relay IR [1] and applying it to Arrow data.
> > > >>
> > > >> As the Arrow memory format has now matured I fell like this could be
> > the
> > > >> next step. Is there any plan for this kind of work or are we going
> to
> > > >> allow sub-projects to "go their own way"?
> > > >>
> > > >> Thanks,
> > > >> Paddy
> > > >>
> > > >> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
> > apache.org
> > > )<
> > > >> https://tvm.apache.org/docs/dev/relay_intro.html>
> > > >>
> > > >>
> > >
> >
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Wes McKinney <we...@gmail.com>.
> I might be misunderstanding, but I think Weld [1] is another project targeting the lower level components?
Weld IR is _really_ low level (not an expert, but have read the
papers), see [1] for more
> Also, I think there was a little bit of effort to come up with a common expression representation within C++, but got stalled on whether to use the Gandiva expression descriptions or Flatbuffer
I'm fine to say "let's just use Protobuf". Trying to unify Gandiva's
and Ballista's protobuf defs for serializing expressions and physical
execution plans seems like a reasonable place to start. If some
enterprising person wanted to enable DataFusion to JIT certain
expressions with Gandiva, that would be some good evidence of being on
the right track (sounds like a good capstone project for an advanced
database systems course).
Per Andrew's comment:
> Thus focusing initially on a standard for expressions might be a good way to add value but keep the scope of the effort reasonable
Here I think either we should base the exprs format either on what
Gandiva's done, and if that isn't general enough for some reason it
would be good to see the analysis / reasoning for doing it
differently. If everyone wants to do it a little differently, then we
might implement an alternate serialized interface to Gandiva
eventually.
I don't think we should get into SQL or SQL dialects at this moment,
at least not until we've shown we can create a standard for
lower-level query plans. SQL doesn't provide any indication of how the
query translates into an execution plan, so if the barrier to entry to
build an "engine" is to parse, analyze, optimize, and plan a query,
that is quite high. However, building an engine that knows how to
evaluate exprs and do hash aggregations and joins, that is a much
smaller scope problem.
Note that any "query plan" generated with this system may be in a way
somewhat higher level than what exactly the engine ends up doing. For
example, an engine might decide to push down predicates into scanners
or otherwise modify the precise order of operations.
[1]: https://github.com/weld-project/weld/blob/master/docs/language.md
On Fri, Mar 19, 2021 at 12:37 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> If we want this format to be common to different execution engines then
> it seems like it should represent logical expressions indeed (which may
> be implemented by different physical operators, depending on the
> execution engine). But I'm no expert in the matter.
>
> Regards
>
> Antoine.
>
>
> Le 18/03/2021 à 17:22, Andrew Lamb a écrit :
> > Any higher level physical execution plan most likely needs a way to
> > represent expressions. Thus focusing initially on a standard for
> > expressions might be a good way to add value but keep the scope of the
> > effort reasonable
> >
> > On Thu, Mar 18, 2021 at 11:49 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> I think there might be discussion on two levels of computation, physical
> >> query execution plans, and potentially something "lower level"? When this
> >> has come up in the past, I was a little skeptical of constraining every SDK
> >> to use the same description, so I agree with Wes's point about keeping any
> >> spec open in the short term. Ballista as an opt-in model, does sound like
> >> possibly the right approach.
> >>
> >> I might be misunderstanding, but I think Weld [1] is another project
> >> targeting the lower level components?
> >>
> >> Also, I think there was a little bit of effort to come up with a common
> >> expression representation within C++, but got stalled on whether to use the
> >> Gandiva expression descriptions or Flatbuffers, I can't seem to find the
> >> thread/JIRA/discussion on this. I'll try to look some more this evening.
> >>
> >> [1] https://github.com/weld-project/weld
> >>
> >> On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
> >>
> >>> I'm interested in providing some path to make this extensible. To pick an
> >>> example, suppose the user wants to compute the first k principle
> >>> components. We've talked [1] about the possibility of incorporating
> >> richer
> >>> communication semantics in Ballista (a la MPI sub-communicators) and
> >>> numerical algorithms such as PCA would benefit. Those specific algorithms
> >>> wouldn't belong in Arrow or Ballista core, but I think there's an
> >>> opportunity for plugins to offer this sort of capability and it would be
> >>> lovely if the language-independent protocol could call them. Do you see a
> >>> good way to do this via ballista.proto?
> >>>
> >>> [1] https://github.com/ballista-compute/ballista/issues/303
> >>>
> >>> Andy Grove <an...@gmail.com> writes:
> >>>
> >>>> Hi Paddy,
> >>>>
> >>>> Thanks for raising this.
> >>>>
> >>>> Ballista defines computations using protobuf [1] to describe logical
> >> and
> >>>> physical query plans, which consist of operators and expressions. It is
> >>>> actually based on the Gandiva protobuf [2] for describing expressions.
> >>>>
> >>>> I see a lot of value in standardizing some of this across
> >>> implementations.
> >>>> Ballista is essentially becoming a distributed scheduler for Arrow and
> >>> can
> >>>> work with any implementation that supports this protobuf definition of
> >>>> query plans.
> >>>>
> >>>> It would also make it easier to embed C++ in Rust, or Rust in C++,
> >> having
> >>>> this common IR, so I would be all for having something like this as an
> >>>> Arrow specification.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Andy.
> >>>>
> >>>> [1]
> >>>>
> >>>
> >> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> >>>> [2]
> >>>>
> >>>
> >> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> >>>>
> >>>>
> >>>> On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I do not have a computer science background so I may not be asking
> >> this
> >>> in
> >>>>> the correct way or using the correct terminology but I wonder if we
> >> can
> >>>>> achieve some level of standardization when describing computation over
> >>>>> Arrow data.
> >>>>>
> >>>>> At the moment on the Rust side DataFusion clearly has a way to
> >> describe
> >>>>> computation, I believe that Ballista adds the ability to serialize
> >> this
> >>> to
> >>>>> allow distributed computation. On the C++ side work is starting on a
> >>>>> similar query engine and we already have Gandiva. Is there an
> >>> opportunity
> >>>>> to define a kind of IR for computation over Arrow data that could be
> >>>>> adopted across implementations?
> >>>>>
> >>>>> In this case DataFusion could easily incorporate Gandiva to generate
> >>>>> optimized compute kernels if they were using the same IR to describe
> >>>>> computation. Applications built on Arrow could "describe" computation
> >>> in
> >>>>> any language and take advantage or innovations across the community,
> >>> adding
> >>>>> this to Arrow's zero copy data sharing could be a game changer in my
> >>> mind.
> >>>>> I'm not someone who knows enough to drive this forward but I obviously
> >>>>> would like to get involved. For some time I was playing around with
> >>> using
> >>>>> TVM's relay IR [1] and applying it to Arrow data.
> >>>>>
> >>>>> As the Arrow memory format has now matured I fell like this could be
> >> the
> >>>>> next step. Is there any plan for this kind of work or are we going to
> >>>>> allow sub-projects to "go their own way"?
> >>>>>
> >>>>> Thanks,
> >>>>> Paddy
> >>>>>
> >>>>> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
> >> apache.org
> >>> )<
> >>>>> https://tvm.apache.org/docs/dev/relay_intro.html>
> >>>>>
> >>>>>
> >>>
> >>
> >
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Antoine Pitrou <an...@python.org>.
If we want this format to be common to different execution engines then
it seems like it should represent logical expressions indeed (which may
be implemented by different physical operators, depending on the
execution engine). But I'm no expert in the matter.
Regards
Antoine.
Le 18/03/2021 à 17:22, Andrew Lamb a écrit :
> Any higher level physical execution plan most likely needs a way to
> represent expressions. Thus focusing initially on a standard for
> expressions might be a good way to add value but keep the scope of the
> effort reasonable
>
> On Thu, Mar 18, 2021 at 11:49 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> I think there might be discussion on two levels of computation, physical
>> query execution plans, and potentially something "lower level"? When this
>> has come up in the past, I was a little skeptical of constraining every SDK
>> to use the same description, so I agree with Wes's point about keeping any
>> spec open in the short term. Ballista as an opt-in model, does sound like
>> possibly the right approach.
>>
>> I might be misunderstanding, but I think Weld [1] is another project
>> targeting the lower level components?
>>
>> Also, I think there was a little bit of effort to come up with a common
>> expression representation within C++, but got stalled on whether to use the
>> Gandiva expression descriptions or Flatbuffers, I can't seem to find the
>> thread/JIRA/discussion on this. I'll try to look some more this evening.
>>
>> [1] https://github.com/weld-project/weld
>>
>> On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
>>
>>> I'm interested in providing some path to make this extensible. To pick an
>>> example, suppose the user wants to compute the first k principle
>>> components. We've talked [1] about the possibility of incorporating
>> richer
>>> communication semantics in Ballista (a la MPI sub-communicators) and
>>> numerical algorithms such as PCA would benefit. Those specific algorithms
>>> wouldn't belong in Arrow or Ballista core, but I think there's an
>>> opportunity for plugins to offer this sort of capability and it would be
>>> lovely if the language-independent protocol could call them. Do you see a
>>> good way to do this via ballista.proto?
>>>
>>> [1] https://github.com/ballista-compute/ballista/issues/303
>>>
>>> Andy Grove <an...@gmail.com> writes:
>>>
>>>> Hi Paddy,
>>>>
>>>> Thanks for raising this.
>>>>
>>>> Ballista defines computations using protobuf [1] to describe logical
>> and
>>>> physical query plans, which consist of operators and expressions. It is
>>>> actually based on the Gandiva protobuf [2] for describing expressions.
>>>>
>>>> I see a lot of value in standardizing some of this across
>>> implementations.
>>>> Ballista is essentially becoming a distributed scheduler for Arrow and
>>> can
>>>> work with any implementation that supports this protobuf definition of
>>>> query plans.
>>>>
>>>> It would also make it easier to embed C++ in Rust, or Rust in C++,
>> having
>>>> this common IR, so I would be all for having something like this as an
>>>> Arrow specification.
>>>>
>>>> Thanks,
>>>>
>>>> Andy.
>>>>
>>>> [1]
>>>>
>>>
>> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
>>>> [2]
>>>>
>>>
>> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>>>>
>>>>
>>>> On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I do not have a computer science background so I may not be asking
>> this
>>> in
>>>>> the correct way or using the correct terminology but I wonder if we
>> can
>>>>> achieve some level of standardization when describing computation over
>>>>> Arrow data.
>>>>>
>>>>> At the moment on the Rust side DataFusion clearly has a way to
>> describe
>>>>> computation, I believe that Ballista adds the ability to serialize
>> this
>>> to
>>>>> allow distributed computation. On the C++ side work is starting on a
>>>>> similar query engine and we already have Gandiva. Is there an
>>> opportunity
>>>>> to define a kind of IR for computation over Arrow data that could be
>>>>> adopted across implementations?
>>>>>
>>>>> In this case DataFusion could easily incorporate Gandiva to generate
>>>>> optimized compute kernels if they were using the same IR to describe
>>>>> computation. Applications built on Arrow could "describe" computation
>>> in
>>>>> any language and take advantage or innovations across the community,
>>> adding
>>>>> this to Arrow's zero copy data sharing could be a game changer in my
>>> mind.
>>>>> I'm not someone who knows enough to drive this forward but I obviously
>>>>> would like to get involved. For some time I was playing around with
>>> using
>>>>> TVM's relay IR [1] and applying it to Arrow data.
>>>>>
>>>>> As the Arrow memory format has now matured I fell like this could be
>> the
>>>>> next step. Is there any plan for this kind of work or are we going to
>>>>> allow sub-projects to "go their own way"?
>>>>>
>>>>> Thanks,
>>>>> Paddy
>>>>>
>>>>> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
>> apache.org
>>> )<
>>>>> https://tvm.apache.org/docs/dev/relay_intro.html>
>>>>>
>>>>>
>>>
>>
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Andrew Lamb <al...@influxdata.com>.
Any higher level physical execution plan most likely needs a way to
represent expressions. Thus focusing initially on a standard for
expressions might be a good way to add value but keep the scope of the
effort reasonable
On Thu, Mar 18, 2021 at 11:49 AM Micah Kornfield <em...@gmail.com>
wrote:
> I think there might be discussion on two levels of computation, physical
> query execution plans, and potentially something "lower level"? When this
> has come up in the past, I was a little skeptical of constraining every SDK
> to use the same description, so I agree with Wes's point about keeping any
> spec open in the short term. Ballista as an opt-in model, does sound like
> possibly the right approach.
>
> I might be misunderstanding, but I think Weld [1] is another project
> targeting the lower level components?
>
> Also, I think there was a little bit of effort to come up with a common
> expression representation within C++, but got stalled on whether to use the
> Gandiva expression descriptions or Flatbuffers, I can't seem to find the
> thread/JIRA/discussion on this. I'll try to look some more this evening.
>
> [1] https://github.com/weld-project/weld
>
> On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
>
> > I'm interested in providing some path to make this extensible. To pick an
> > example, suppose the user wants to compute the first k principle
> > components. We've talked [1] about the possibility of incorporating
> richer
> > communication semantics in Ballista (a la MPI sub-communicators) and
> > numerical algorithms such as PCA would benefit. Those specific algorithms
> > wouldn't belong in Arrow or Ballista core, but I think there's an
> > opportunity for plugins to offer this sort of capability and it would be
> > lovely if the language-independent protocol could call them. Do you see a
> > good way to do this via ballista.proto?
> >
> > [1] https://github.com/ballista-compute/ballista/issues/303
> >
> > Andy Grove <an...@gmail.com> writes:
> >
> > > Hi Paddy,
> > >
> > > Thanks for raising this.
> > >
> > > Ballista defines computations using protobuf [1] to describe logical
> and
> > > physical query plans, which consist of operators and expressions. It is
> > > actually based on the Gandiva protobuf [2] for describing expressions.
> > >
> > > I see a lot of value in standardizing some of this across
> > implementations.
> > > Ballista is essentially becoming a distributed scheduler for Arrow and
> > can
> > > work with any implementation that supports this protobuf definition of
> > > query plans.
> > >
> > > It would also make it easier to embed C++ in Rust, or Rust in C++,
> having
> > > this common IR, so I would be all for having something like this as an
> > > Arrow specification.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > > [1]
> > >
> >
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> > > [2]
> > >
> >
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> > >
> > >
> > > On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I do not have a computer science background so I may not be asking
> this
> > in
> > >> the correct way or using the correct terminology but I wonder if we
> can
> > >> achieve some level of standardization when describing computation over
> > >> Arrow data.
> > >>
> > >> At the moment on the Rust side DataFusion clearly has a way to
> describe
> > >> computation, I believe that Ballista adds the ability to serialize
> this
> > to
> > >> allow distributed computation. On the C++ side work is starting on a
> > >> similar query engine and we already have Gandiva. Is there an
> > opportunity
> > >> to define a kind of IR for computation over Arrow data that could be
> > >> adopted across implementations?
> > >>
> > >> In this case DataFusion could easily incorporate Gandiva to generate
> > >> optimized compute kernels if they were using the same IR to describe
> > >> computation. Applications built on Arrow could "describe" computation
> > in
> > >> any language and take advantage or innovations across the community,
> > adding
> > >> this to Arrow's zero copy data sharing could be a game changer in my
> > mind.
> > >> I'm not someone who knows enough to drive this forward but I obviously
> > >> would like to get involved. For some time I was playing around with
> > using
> > >> TVM's relay IR [1] and applying it to Arrow data.
> > >>
> > >> As the Arrow memory format has now matured I fell like this could be
> the
> > >> next step. Is there any plan for this kind of work or are we going to
> > >> allow sub-projects to "go their own way"?
> > >>
> > >> Thanks,
> > >> Paddy
> > >>
> > >> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
> apache.org
> > )<
> > >> https://tvm.apache.org/docs/dev/relay_intro.html>
> > >>
> > >>
> >
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Micah Kornfield <em...@gmail.com>.
I think there might be discussion on two levels of computation, physical
query execution plans, and potentially something "lower level"? When this
has come up in the past, I was a little skeptical of constraining every SDK
to use the same description, so I agree with Wes's point about keeping any
spec open in the short term. Ballista as an opt-in model, does sound like
possibly the right approach.
I might be misunderstanding, but I think Weld [1] is another project
targeting the lower level components?
Also, I think there was a little bit of effort to come up with a common
expression representation within C++, but got stalled on whether to use the
Gandiva expression descriptions or Flatbuffers, I can't seem to find the
thread/JIRA/discussion on this. I'll try to look some more this evening.
[1] https://github.com/weld-project/weld
On Thu, Mar 18, 2021 at 7:53 AM Jed Brown <je...@jedbrown.org> wrote:
> I'm interested in providing some path to make this extensible. To pick an
> example, suppose the user wants to compute the first k principle
> components. We've talked [1] about the possibility of incorporating richer
> communication semantics in Ballista (a la MPI sub-communicators) and
> numerical algorithms such as PCA would benefit. Those specific algorithms
> wouldn't belong in Arrow or Ballista core, but I think there's an
> opportunity for plugins to offer this sort of capability and it would be
> lovely if the language-independent protocol could call them. Do you see a
> good way to do this via ballista.proto?
>
> [1] https://github.com/ballista-compute/ballista/issues/303
>
> Andy Grove <an...@gmail.com> writes:
>
> > Hi Paddy,
> >
> > Thanks for raising this.
> >
> > Ballista defines computations using protobuf [1] to describe logical and
> > physical query plans, which consist of operators and expressions. It is
> > actually based on the Gandiva protobuf [2] for describing expressions.
> >
> > I see a lot of value in standardizing some of this across
> implementations.
> > Ballista is essentially becoming a distributed scheduler for Arrow and
> can
> > work with any implementation that supports this protobuf definition of
> > query plans.
> >
> > It would also make it easier to embed C++ in Rust, or Rust in C++, having
> > this common IR, so I would be all for having something like this as an
> > Arrow specification.
> >
> > Thanks,
> >
> > Andy.
> >
> > [1]
> >
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> > [2]
> >
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> >
> >
> > On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> wrote:
> >
> >> Hi All,
> >>
> >> I do not have a computer science background so I may not be asking this
> in
> >> the correct way or using the correct terminology but I wonder if we can
> >> achieve some level of standardization when describing computation over
> >> Arrow data.
> >>
> >> At the moment on the Rust side DataFusion clearly has a way to describe
> >> computation, I believe that Ballista adds the ability to serialize this
> to
> >> allow distributed computation. On the C++ side work is starting on a
> >> similar query engine and we already have Gandiva. Is there an
> opportunity
> >> to define a kind of IR for computation over Arrow data that could be
> >> adopted across implementations?
> >>
> >> In this case DataFusion could easily incorporate Gandiva to generate
> >> optimized compute kernels if they were using the same IR to describe
> >> computation. Applications built on Arrow could "describe" computation
> in
> >> any language and take advantage or innovations across the community,
> adding
> >> this to Arrow's zero copy data sharing could be a game changer in my
> mind.
> >> I'm not someone who knows enough to drive this forward but I obviously
> >> would like to get involved. For some time I was playing around with
> using
> >> TVM's relay IR [1] and applying it to Arrow data.
> >>
> >> As the Arrow memory format has now matured I fell like this could be the
> >> next step. Is there any plan for this kind of work or are we going to
> >> allow sub-projects to "go their own way"?
> >>
> >> Thanks,
> >> Paddy
> >>
> >> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (apache.org
> )<
> >> https://tvm.apache.org/docs/dev/relay_intro.html>
> >>
> >>
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Jed Brown <je...@jedbrown.org>.
I'm interested in providing some path to make this extensible. To pick an example, suppose the user wants to compute the first k principle components. We've talked [1] about the possibility of incorporating richer communication semantics in Ballista (a la MPI sub-communicators) and numerical algorithms such as PCA would benefit. Those specific algorithms wouldn't belong in Arrow or Ballista core, but I think there's an opportunity for plugins to offer this sort of capability and it would be lovely if the language-independent protocol could call them. Do you see a good way to do this via ballista.proto?
[1] https://github.com/ballista-compute/ballista/issues/303
Andy Grove <an...@gmail.com> writes:
> Hi Paddy,
>
> Thanks for raising this.
>
> Ballista defines computations using protobuf [1] to describe logical and
> physical query plans, which consist of operators and expressions. It is
> actually based on the Gandiva protobuf [2] for describing expressions.
>
> I see a lot of value in standardizing some of this across implementations.
> Ballista is essentially becoming a distributed scheduler for Arrow and can
> work with any implementation that supports this protobuf definition of
> query plans.
>
> It would also make it easier to embed C++ in Rust, or Rust in C++, having
> this common IR, so I would be all for having something like this as an
> Arrow specification.
>
> Thanks,
>
> Andy.
>
> [1]
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> [2]
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>
>
> On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com> wrote:
>
>> Hi All,
>>
>> I do not have a computer science background so I may not be asking this in
>> the correct way or using the correct terminology but I wonder if we can
>> achieve some level of standardization when describing computation over
>> Arrow data.
>>
>> At the moment on the Rust side DataFusion clearly has a way to describe
>> computation, I believe that Ballista adds the ability to serialize this to
>> allow distributed computation. On the C++ side work is starting on a
>> similar query engine and we already have Gandiva. Is there an opportunity
>> to define a kind of IR for computation over Arrow data that could be
>> adopted across implementations?
>>
>> In this case DataFusion could easily incorporate Gandiva to generate
>> optimized compute kernels if they were using the same IR to describe
>> computation. Applications built on Arrow could "describe" computation in
>> any language and take advantage or innovations across the community, adding
>> this to Arrow's zero copy data sharing could be a game changer in my mind.
>> I'm not someone who knows enough to drive this forward but I obviously
>> would like to get involved. For some time I was playing around with using
>> TVM's relay IR [1] and applying it to Arrow data.
>>
>> As the Arrow memory format has now matured I fell like this could be the
>> next step. Is there any plan for this kind of work or are we going to
>> allow sub-projects to "go their own way"?
>>
>> Thanks,
>> Paddy
>>
>> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (apache.org)<
>> https://tvm.apache.org/docs/dev/relay_intro.html>
>>
>>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Brian Hulette <bh...@apache.org>.
I agree this would be a great development. It would also be useful for
leveraging compute engines from JS via wasm.
I've thought about something like this in the context of multi-language
relational workloads in Apache Beam, mostly just leading me to wonder if
something like it already exists. But so far I haven't found it.
On Thu, Mar 18, 2021 at 7:39 AM Wes McKinney <we...@gmail.com> wrote:
> I completely agree with developing a common “query protocol” or “physical
> execution plan” IR + serialization scheme inside Apache Arrow. It may take
> some time to stabilize so we should try to avoid being hasty in closing it
> to change until more time has elapsed to allow requirements to percolate.
>
> On Thu, Mar 18, 2021 at 8:17 AM Andy Grove <an...@gmail.com> wrote:
>
> > Hi Paddy,
> >
> > Thanks for raising this.
> >
> > Ballista defines computations using protobuf [1] to describe logical and
> > physical query plans, which consist of operators and expressions. It is
> > actually based on the Gandiva protobuf [2] for describing expressions.
> >
> > I see a lot of value in standardizing some of this across
> implementations.
> > Ballista is essentially becoming a distributed scheduler for Arrow and
> can
> > work with any implementation that supports this protobuf definition of
> > query plans.
> >
> > It would also make it easier to embed C++ in Rust, or Rust in C++, having
> > this common IR, so I would be all for having something like this as an
> > Arrow specification.
> >
> > Thanks,
> >
> > Andy.
> >
> > [1]
> >
> >
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> > [2]
> >
> >
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
> >
> >
> > On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I do not have a computer science background so I may not be asking this
> > in
> > > the correct way or using the correct terminology but I wonder if we can
> > > achieve some level of standardization when describing computation over
> > > Arrow data.
> > >
> > > At the moment on the Rust side DataFusion clearly has a way to describe
> > > computation, I believe that Ballista adds the ability to serialize this
> > to
> > > allow distributed computation. On the C++ side work is starting on a
> > > similar query engine and we already have Gandiva. Is there an
> > opportunity
> > > to define a kind of IR for computation over Arrow data that could be
> > > adopted across implementations?
> > >
> > > In this case DataFusion could easily incorporate Gandiva to generate
> > > optimized compute kernels if they were using the same IR to describe
> > > computation. Applications built on Arrow could "describe" computation
> in
> > > any language and take advantage or innovations across the community,
> > adding
> > > this to Arrow's zero copy data sharing could be a game changer in my
> > mind.
> > > I'm not someone who knows enough to drive this forward but I obviously
> > > would like to get involved. For some time I was playing around with
> > using
> > > TVM's relay IR [1] and applying it to Arrow data.
> > >
> > > As the Arrow memory format has now matured I fell like this could be
> the
> > > next step. Is there any plan for this kind of work or are we going to
> > > allow sub-projects to "go their own way"?
> > >
> > > Thanks,
> > > Paddy
> > >
> > > [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (
> apache.org
> > )<
> > > https://tvm.apache.org/docs/dev/relay_intro.html>
> > >
> > >
> >
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Wes McKinney <we...@gmail.com>.
I completely agree with developing a common “query protocol” or “physical
execution plan” IR + serialization scheme inside Apache Arrow. It may take
some time to stabilize so we should try to avoid being hasty in closing it
to change until more time has elapsed to allow requirements to percolate.
On Thu, Mar 18, 2021 at 8:17 AM Andy Grove <an...@gmail.com> wrote:
> Hi Paddy,
>
> Thanks for raising this.
>
> Ballista defines computations using protobuf [1] to describe logical and
> physical query plans, which consist of operators and expressions. It is
> actually based on the Gandiva protobuf [2] for describing expressions.
>
> I see a lot of value in standardizing some of this across implementations.
> Ballista is essentially becoming a distributed scheduler for Arrow and can
> work with any implementation that supports this protobuf definition of
> query plans.
>
> It would also make it easier to embed C++ in Rust, or Rust in C++, having
> this common IR, so I would be all for having something like this as an
> Arrow specification.
>
> Thanks,
>
> Andy.
>
> [1]
>
> https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
> [2]
>
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
>
>
> On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com>
> wrote:
>
> > Hi All,
> >
> > I do not have a computer science background so I may not be asking this
> in
> > the correct way or using the correct terminology but I wonder if we can
> > achieve some level of standardization when describing computation over
> > Arrow data.
> >
> > At the moment on the Rust side DataFusion clearly has a way to describe
> > computation, I believe that Ballista adds the ability to serialize this
> to
> > allow distributed computation. On the C++ side work is starting on a
> > similar query engine and we already have Gandiva. Is there an
> opportunity
> > to define a kind of IR for computation over Arrow data that could be
> > adopted across implementations?
> >
> > In this case DataFusion could easily incorporate Gandiva to generate
> > optimized compute kernels if they were using the same IR to describe
> > computation. Applications built on Arrow could "describe" computation in
> > any language and take advantage or innovations across the community,
> adding
> > this to Arrow's zero copy data sharing could be a game changer in my
> mind.
> > I'm not someone who knows enough to drive this forward but I obviously
> > would like to get involved. For some time I was playing around with
> using
> > TVM's relay IR [1] and applying it to Arrow data.
> >
> > As the Arrow memory format has now matured I fell like this could be the
> > next step. Is there any plan for this kind of work or are we going to
> > allow sub-projects to "go their own way"?
> >
> > Thanks,
> > Paddy
> >
> > [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (apache.org
> )<
> > https://tvm.apache.org/docs/dev/relay_intro.html>
> >
> >
>
Re: [DISCUSS] How to describe computation on Arrow data?
Posted by Andy Grove <an...@gmail.com>.
Hi Paddy,
Thanks for raising this.
Ballista defines computations using protobuf [1] to describe logical and
physical query plans, which consist of operators and expressions. It is
actually based on the Gandiva protobuf [2] for describing expressions.
I see a lot of value in standardizing some of this across implementations.
Ballista is essentially becoming a distributed scheduler for Arrow and can
work with any implementation that supports this protobuf definition of
query plans.
It would also make it easier to embed C++ in Rust, or Rust in C++, having
this common IR, so I would be all for having something like this as an
Arrow specification.
Thanks,
Andy.
[1]
https://github.com/ballista-compute/ballista/blob/main/rust/core/proto/ballista.proto
[2]
https://github.com/apache/arrow/blob/master/cpp/src/gandiva/proto/Types.proto
On Thu, Mar 18, 2021 at 7:40 AM paddy horan <pa...@hotmail.com> wrote:
> Hi All,
>
> I do not have a computer science background so I may not be asking this in
> the correct way or using the correct terminology but I wonder if we can
> achieve some level of standardization when describing computation over
> Arrow data.
>
> At the moment on the Rust side DataFusion clearly has a way to describe
> computation, I believe that Ballista adds the ability to serialize this to
> allow distributed computation. On the C++ side work is starting on a
> similar query engine and we already have Gandiva. Is there an opportunity
> to define a kind of IR for computation over Arrow data that could be
> adopted across implementations?
>
> In this case DataFusion could easily incorporate Gandiva to generate
> optimized compute kernels if they were using the same IR to describe
> computation. Applications built on Arrow could "describe" computation in
> any language and take advantage or innovations across the community, adding
> this to Arrow's zero copy data sharing could be a game changer in my mind.
> I'm not someone who knows enough to drive this forward but I obviously
> would like to get involved. For some time I was playing around with using
> TVM's relay IR [1] and applying it to Arrow data.
>
> As the Arrow memory format has now matured I fell like this could be the
> next step. Is there any plan for this kind of work or are we going to
> allow sub-projects to "go their own way"?
>
> Thanks,
> Paddy
>
> [1] - Introduction to Relay IR - tvm 0.8.dev0 documentation (apache.org)<
> https://tvm.apache.org/docs/dev/relay_intro.html>
>
>