You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ravindra Pindikura <ra...@dremio.com> on 2019/02/13 07:34:24 UTC
Compute kernels and Gandiva operators
Hi,
I was looking at the recent checkin for arrow kernels, and started to think of how they would work alongside Gandiva.
Here are my thoughts :
1. Gandiva already has two high-level operators namely project and filter, with runtime code generation
* It already supports 100s of functions (eg. a+b, a > b), which can be combined into expressions (eg. a+b > c && a +b < d) for each of the operators and we’ll likely continue to add more of them.
* it works on one record batch at a time - consumes a record batch, and produces a record batch.
* The operators can be inter-linked (eg. Project -> filter -> project) to build a pipeline.
* we may build additional operators in the future which could benefit from code generation (eg. Impala uses code generation when parsing Avro files).
2. Arrow Kernels
a. support project/filter operators
Useful for functions where there is no benefit with code generation, or where code generation can be skipped over (eager evaluation).
b. Support for additional operators like aggregates
How do we combine and link the gandiva operators and the kernels ? For eg. It would be nice to have a pipeline with scan (read from source), project (expression on column), filter (extract rows), and aggregate (sum on the extracted column).
To do this, I think we would need to be able build a pipeline with high level operators that move along data one record batch at a time :
- source operator which only produces record-batches (maybe, csv reader)
- intermediate operators that can produce/consume record-batches (maybe, gandiva project operator)
- terminal operators that emit the final output (from the end of the pipeline) when there is nothing left to consume (maybe, SumKernel)
Are we thinking along these lines ?
Thanks & regards,
Ravindra.
Re: Compute kernels and Gandiva operators
Posted by Ravindra Pindikura <ra...@dremio.com>.
Thanks, Wes and Francois.
> On Feb 13, 2019, at 10:16 PM, Francois Saint-Jacques <fs...@gmail.com> wrote:
>
> Hi,
>
> I also agree that we should follow a model similar to what you propose. I
> think the plan is, correct me if I'm wrong Wes, to write the logical plan
> operators, then write a small execution engine prototype and produce a
> proper design document out of this experiment. There's also a placeholder
> ticket: https://issues.apache.org/jira/browse/ARROW-4333
>
> François
>
> On Wed, Feb 13, 2019 at 9:54 AM Wes McKinney <we...@gmail.com> wrote:
>
>> hi Ravindra,
>>
>>
>> On Wed, Feb 13, 2019 at 1:34 AM Ravindra Pindikura <ra...@dremio.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I was looking at the recent checkin for arrow kernels, and started to
>> think of how they would work alongside Gandiva.
>>>
>>> Here are my thoughts :
>>>
>>> 1. Gandiva already has two high-level operators namely project and
>> filter, with runtime code generation
>>>
>>> * It already supports 100s of functions (eg. a+b, a > b), which can be
>> combined into expressions (eg. a+b > c && a +b < d) for each of the
>> operators and we’ll likely continue to add more of them.
>>> * it works on one record batch at a time - consumes a record batch, and
>> produces a record batch.
>>> * The operators can be inter-linked (eg. Project -> filter -> project)
>> to build a pipeline.
>>> * we may build additional operators in the future which could benefit
>> from code generation (eg. Impala uses code generation when parsing Avro
>> files).
>>>
>>> 2. Arrow Kernels
>>>
>>> a. support project/filter operators
>>>
>>> Useful for functions where there is no benefit with code generation, or
>> where code generation can be skipped over (eager evaluation).
>>>
>>> b. Support for additional operators like aggregates
>>>
>>>
>>> How do we combine and link the gandiva operators and the kernels ? For
>> eg. It would be nice to have a pipeline with scan (read from source),
>> project (expression on column), filter (extract rows), and aggregate (sum
>> on the extracted column).
>>>
>>
>> I'm just starting on a project now to enable logical expressions to be
>> built with a C++ API that are formed from a superset of relationship
>> algebra. In the past, I have already built a fairly deep system for
>> this (https://github.com/ibis-project/ibis) that captures SQL
>> semantics while also permitting other kinds of analytical
>> functionality that spills outside of SQL. As soon as I cobble together
>> a prototype I'll be interested in lots of feedback since this is a
>> system that we'll use for many years
>>
>> My idea of the interplay with Gandiva is that expressions will be
>> "lowered" to physical execution operators such as:
>>
>> * Projection (with/without filters)
>> * Join
>> * Hash aggregate (with/without filters)
>> * Sort
>> * Time series operations (resampling / binning, etc.)
>>
>> Gandiva therefore serves as a subgraph compiler for constituent parts
>> of these operations . During the lowering process, operator subgraphs
>> will need to be recognized as "able to be compiled with Gandiva". For
>> example
>>
>> * log(table.a + table.b) + 1 can be compiled with gandiva
>> * udf(table.a), where "udf" is a user-defined function written in
>> Python, say, cannot
>>
>>> To do this, I think we would need to be able build a pipeline with high
>> level operators that move along data one record batch at a time :
>>> - source operator which only produces record-batches (maybe, csv reader)
>>> - intermediate operators that can produce/consume record-batches (maybe,
>> gandiva project operator)
>>> - terminal operators that emit the final output (from the end of the
>> pipeline) when there is nothing left to consume (maybe, SumKernel)
>>>
>>> Are we thinking along these lines ?
>>
>> Yes, that sounds right.
>>
>> We don't have a "dataset" framework yet for source operators (CSV,
>> JSON, and Parquet will be the first ones). I have planned to write a
>> requirements document about this project as we have the machinery in
>> place to e.g. read Parquet and CSV files but without a unified
>> abstract dataset API
>>
>> - Wes
>>
>>>
>>> Thanks & regards,
>>> Ravindra.
>>
Re: Compute kernels and Gandiva operators
Posted by Francois Saint-Jacques <fs...@gmail.com>.
Hi,
I also agree that we should follow a model similar to what you propose. I
think the plan is, correct me if I'm wrong Wes, to write the logical plan
operators, then write a small execution engine prototype and produce a
proper design document out of this experiment. There's also a placeholder
ticket: https://issues.apache.org/jira/browse/ARROW-4333
François
On Wed, Feb 13, 2019 at 9:54 AM Wes McKinney <we...@gmail.com> wrote:
> hi Ravindra,
>
>
> On Wed, Feb 13, 2019 at 1:34 AM Ravindra Pindikura <ra...@dremio.com>
> wrote:
> >
> > Hi,
> >
> > I was looking at the recent checkin for arrow kernels, and started to
> think of how they would work alongside Gandiva.
> >
> > Here are my thoughts :
> >
> > 1. Gandiva already has two high-level operators namely project and
> filter, with runtime code generation
> >
> > * It already supports 100s of functions (eg. a+b, a > b), which can be
> combined into expressions (eg. a+b > c && a +b < d) for each of the
> operators and we’ll likely continue to add more of them.
> > * it works on one record batch at a time - consumes a record batch, and
> produces a record batch.
> > * The operators can be inter-linked (eg. Project -> filter -> project)
> to build a pipeline.
> > * we may build additional operators in the future which could benefit
> from code generation (eg. Impala uses code generation when parsing Avro
> files).
> >
> > 2. Arrow Kernels
> >
> > a. support project/filter operators
> >
> > Useful for functions where there is no benefit with code generation, or
> where code generation can be skipped over (eager evaluation).
> >
> > b. Support for additional operators like aggregates
> >
> >
> > How do we combine and link the gandiva operators and the kernels ? For
> eg. It would be nice to have a pipeline with scan (read from source),
> project (expression on column), filter (extract rows), and aggregate (sum
> on the extracted column).
> >
>
> I'm just starting on a project now to enable logical expressions to be
> built with a C++ API that are formed from a superset of relationship
> algebra. In the past, I have already built a fairly deep system for
> this (https://github.com/ibis-project/ibis) that captures SQL
> semantics while also permitting other kinds of analytical
> functionality that spills outside of SQL. As soon as I cobble together
> a prototype I'll be interested in lots of feedback since this is a
> system that we'll use for many years
>
> My idea of the interplay with Gandiva is that expressions will be
> "lowered" to physical execution operators such as:
>
> * Projection (with/without filters)
> * Join
> * Hash aggregate (with/without filters)
> * Sort
> * Time series operations (resampling / binning, etc.)
>
> Gandiva therefore serves as a subgraph compiler for constituent parts
> of these operations . During the lowering process, operator subgraphs
> will need to be recognized as "able to be compiled with Gandiva". For
> example
>
> * log(table.a + table.b) + 1 can be compiled with gandiva
> * udf(table.a), where "udf" is a user-defined function written in
> Python, say, cannot
>
> > To do this, I think we would need to be able build a pipeline with high
> level operators that move along data one record batch at a time :
> > - source operator which only produces record-batches (maybe, csv reader)
> > - intermediate operators that can produce/consume record-batches (maybe,
> gandiva project operator)
> > - terminal operators that emit the final output (from the end of the
> pipeline) when there is nothing left to consume (maybe, SumKernel)
> >
> > Are we thinking along these lines ?
>
> Yes, that sounds right.
>
> We don't have a "dataset" framework yet for source operators (CSV,
> JSON, and Parquet will be the first ones). I have planned to write a
> requirements document about this project as we have the machinery in
> place to e.g. read Parquet and CSV files but without a unified
> abstract dataset API
>
> - Wes
>
> >
> > Thanks & regards,
> > Ravindra.
>
Re: Compute kernels and Gandiva operators
Posted by Wes McKinney <we...@gmail.com>.
hi Ravindra,
On Wed, Feb 13, 2019 at 1:34 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>
> Hi,
>
> I was looking at the recent checkin for arrow kernels, and started to think of how they would work alongside Gandiva.
>
> Here are my thoughts :
>
> 1. Gandiva already has two high-level operators namely project and filter, with runtime code generation
>
> * It already supports 100s of functions (eg. a+b, a > b), which can be combined into expressions (eg. a+b > c && a +b < d) for each of the operators and we’ll likely continue to add more of them.
> * it works on one record batch at a time - consumes a record batch, and produces a record batch.
> * The operators can be inter-linked (eg. Project -> filter -> project) to build a pipeline.
> * we may build additional operators in the future which could benefit from code generation (eg. Impala uses code generation when parsing Avro files).
>
> 2. Arrow Kernels
>
> a. support project/filter operators
>
> Useful for functions where there is no benefit with code generation, or where code generation can be skipped over (eager evaluation).
>
> b. Support for additional operators like aggregates
>
>
> How do we combine and link the gandiva operators and the kernels ? For eg. It would be nice to have a pipeline with scan (read from source), project (expression on column), filter (extract rows), and aggregate (sum on the extracted column).
>
I'm just starting on a project now to enable logical expressions to be
built with a C++ API that are formed from a superset of relationship
algebra. In the past, I have already built a fairly deep system for
this (https://github.com/ibis-project/ibis) that captures SQL
semantics while also permitting other kinds of analytical
functionality that spills outside of SQL. As soon as I cobble together
a prototype I'll be interested in lots of feedback since this is a
system that we'll use for many years
My idea of the interplay with Gandiva is that expressions will be
"lowered" to physical execution operators such as:
* Projection (with/without filters)
* Join
* Hash aggregate (with/without filters)
* Sort
* Time series operations (resampling / binning, etc.)
Gandiva therefore serves as a subgraph compiler for constituent parts
of these operations . During the lowering process, operator subgraphs
will need to be recognized as "able to be compiled with Gandiva". For
example
* log(table.a + table.b) + 1 can be compiled with gandiva
* udf(table.a), where "udf" is a user-defined function written in
Python, say, cannot
> To do this, I think we would need to be able build a pipeline with high level operators that move along data one record batch at a time :
> - source operator which only produces record-batches (maybe, csv reader)
> - intermediate operators that can produce/consume record-batches (maybe, gandiva project operator)
> - terminal operators that emit the final output (from the end of the pipeline) when there is nothing left to consume (maybe, SumKernel)
>
> Are we thinking along these lines ?
Yes, that sounds right.
We don't have a "dataset" framework yet for source operators (CSV,
JSON, and Parquet will be the first ones). I have planned to write a
requirements document about this project as we have the machinery in
place to e.g. read Parquet and CSV files but without a unified
abstract dataset API
- Wes
>
> Thanks & regards,
> Ravindra.