You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andy Grove <an...@gmail.com> on 2019/01/04 14:39:17 UTC
Rust bindings for Gandiva
Now that the Rust implementation of Arrow is maturing, I'm interested in
having bindings for Gandiva for query execution, rather than duplicating
this in Rust.
I will likely start looking at this soon but wanted to see if anyone else
here is particularly interested in this area of functionality?
Thanks,
Andy.
Re: Rust bindings for Gandiva
Posted by Wes McKinney <we...@gmail.com>.
I have a patch up for Gandiva on Windows:
https://github.com/apache/arrow/pull/3295
It would be great to have the subgraph compiler available in Rust. You
should decide whether you want to require a C++ library that depends
on LLVM into all applications that use Rust and Arrow and need to
compute some expressions. It might be useful to have a mix of
Rust-native analytics and the option to utilize the LLVM compiler
On Fri, Jan 4, 2019 at 12:54 PM paddy horan <pa...@hotmail.com> wrote:
>
> Hey Andy,
>
> I am very interested in this, I’m also looking into adding explicit SIMD to our existing “array_ops”.
>
> Maybe we can plan out what is needed on the developer wiki so that we can all help out where we are able.
>
> I’ve seen it mentioned here and there but what it the current state of gandiva on windows? I’m willing to help where I can but I’m not very experienced with C++/cmake.
>
> P
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Andy Grove <an...@gmail.com>
> Sent: Friday, January 4, 2019 9:39 AM
> To: dev@arrow.apache.org
> Subject: Rust bindings for Gandiva
>
> Now that the Rust implementation of Arrow is maturing, I'm interested in
> having bindings for Gandiva for query execution, rather than duplicating
> this in Rust.
>
> I will likely start looking at this soon but wanted to see if anyone else
> here is particularly interested in this area of functionality?
>
> Thanks,
>
> Andy.
Re: Rust bindings for Gandiva
Posted by paddy horan <pa...@hotmail.com>.
Hey Andy,
I am very interested in this, I’m also looking into adding explicit SIMD to our existing “array_ops”.
Maybe we can plan out what is needed on the developer wiki so that we can all help out where we are able.
I’ve seen it mentioned here and there but what it the current state of gandiva on windows? I’m willing to help where I can but I’m not very experienced with C++/cmake.
P
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Andy Grove <an...@gmail.com>
Sent: Friday, January 4, 2019 9:39 AM
To: dev@arrow.apache.org
Subject: Rust bindings for Gandiva
Now that the Rust implementation of Arrow is maturing, I'm interested in
having bindings for Gandiva for query execution, rather than duplicating
this in Rust.
I will likely start looking at this soon but wanted to see if anyone else
here is particularly interested in this area of functionality?
Thanks,
Andy.
Re: Rust bindings for Gandiva
Posted by Renjie Liu <li...@gmail.com>.
I agree that this should be a separate project, so that this can be used by
other databases written in rust, not only datafusion. Let's start with an
implementation by binding with gandiva, and build pure rust implementation
later.
On Sat, May 11, 2019 at 10:28 PM Andy Grove <an...@gmail.com> wrote:
> Hi Renjie,
>
> I have not started on this but I would be interested in helping you with
> it.
>
> At a high level I think there are two main parts to this work:
>
> 1. Translating DataFusion expressions to Gandiva protobuf
> 2. Implementing the code to make the native C call to Gandiva
>
> I could help with #1 pretty easily.
>
> I am concerned about the packaging implications of this. I feel quite
> strongly that there should be a "pure Rust" version of DataFusion/Arrow and
> that the Gandiva integration should be opt-in somehow, so maybe this is a
> separate project within the repository, or a feature that can be controlled
> by a feature flag in the Cargo.toml somehow.
>
> Thanks,
>
> Andy.
>
>
>
>
> On Sat, May 11, 2019 at 3:10 AM Renjie Liu <li...@gmail.com>
> wrote:
>
>>
>> Hi:
>> @Andy Grove <an...@gmail.com> Are you developing this? I'm
>> interested in this and want to join development.
>>
>> On Tue, Jan 8, 2019 at 3:18 PM Praveen Kumar <pr...@dremio.com> wrote:
>>
>>> Agree with Wes, the protobuf based interface should be the language
>>> neutral
>>> way to build expressions with Gandiva.
>>>
>>> On Mon, Jan 7, 2019 at 8:30 PM Andy Grove <an...@gmail.com> wrote:
>>>
>>> > This makes sense to me know that I understand a little more about
>>> Gandiva.
>>> > This also fits well with my proposal to donate DataFusion in the other
>>> > thread. DataFusion can manage the overall logical query plan in Rust
>>> and
>>> > potentially delegate some subset of expression evaluation to Gandiva
>>> via
>>> > protobuf.
>>> >
>>> > Thanks,
>>> >
>>> > Andy.
>>> >
>>> > On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <we...@gmail.com>
>>> wrote:
>>> >
>>> > > Gandiva supports a Protobuf-based interface -- this is how Java
>>> > > interacts with it via JNI. Rust could do the same -- that would
>>> > > probably be easier than wrapping the C++ class structure. It would
>>> > > also help drive new feature requirements in the serialized
>>> > > projection/filter expression trees
>>> > >
>>> > > - Wes
>>> > >
>>> > > On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
>>> > > <sz...@gmail.com> wrote:
>>> > > >
>>> > > > I'm not sure, that a binding is a good idea. Both Arrow and Parquet
>>> > > > already have their own rust implementation, and a interfacing with
>>> > > > cpp isn't as easy and straightforward than it is with C. Otherwise
>>> > > > We could simply just maintain bindings for all of the cpp
>>> libraries,
>>> > > > rather than of having a hybrid solution.
>>> > > >
>>> > > > While We could spare the reimplementation of gandiva, it'd make
>>> > > > packaging more complicated and rust development way less
>>> > > > welcoming to new contributors.
>>> > > >
>>> > > > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > > > Now that the Rust implementation of Arrow is maturing, I'm
>>> interested
>>> > > in
>>> > > > > having bindings for Gandiva for query execution, rather than
>>> > > duplicating
>>> > > > > this in Rust.
>>> > > > >
>>> > > > > I will likely start looking at this soon but wanted to see if
>>> anyone
>>> > > else
>>> > > > > here is particularly interested in this area of functionality?
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > Andy.
>>> > > > >
>>> > >
>>> >
>>>
>>
>>
>> --
>> Renjie Liu
>> Software Engineer, MVAD
>>
>
--
Renjie Liu
Software Engineer, MVAD
Re: Rust bindings for Gandiva
Posted by Andy Grove <an...@gmail.com>.
Hi Renjie,
I have not started on this but I would be interested in helping you with
it.
At a high level I think there are two main parts to this work:
1. Translating DataFusion expressions to Gandiva protobuf
2. Implementing the code to make the native C call to Gandiva
I could help with #1 pretty easily.
I am concerned about the packaging implications of this. I feel quite
strongly that there should be a "pure Rust" version of DataFusion/Arrow and
that the Gandiva integration should be opt-in somehow, so maybe this is a
separate project within the repository, or a feature that can be controlled
by a feature flag in the Cargo.toml somehow.
Thanks,
Andy.
On Sat, May 11, 2019 at 3:10 AM Renjie Liu <li...@gmail.com> wrote:
>
> Hi:
> @Andy Grove <an...@gmail.com> Are you developing this? I'm
> interested in this and want to join development.
>
> On Tue, Jan 8, 2019 at 3:18 PM Praveen Kumar <pr...@dremio.com> wrote:
>
>> Agree with Wes, the protobuf based interface should be the language
>> neutral
>> way to build expressions with Gandiva.
>>
>> On Mon, Jan 7, 2019 at 8:30 PM Andy Grove <an...@gmail.com> wrote:
>>
>> > This makes sense to me know that I understand a little more about
>> Gandiva.
>> > This also fits well with my proposal to donate DataFusion in the other
>> > thread. DataFusion can manage the overall logical query plan in Rust and
>> > potentially delegate some subset of expression evaluation to Gandiva via
>> > protobuf.
>> >
>> > Thanks,
>> >
>> > Andy.
>> >
>> > On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <we...@gmail.com>
>> wrote:
>> >
>> > > Gandiva supports a Protobuf-based interface -- this is how Java
>> > > interacts with it via JNI. Rust could do the same -- that would
>> > > probably be easier than wrapping the C++ class structure. It would
>> > > also help drive new feature requirements in the serialized
>> > > projection/filter expression trees
>> > >
>> > > - Wes
>> > >
>> > > On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
>> > > <sz...@gmail.com> wrote:
>> > > >
>> > > > I'm not sure, that a binding is a good idea. Both Arrow and Parquet
>> > > > already have their own rust implementation, and a interfacing with
>> > > > cpp isn't as easy and straightforward than it is with C. Otherwise
>> > > > We could simply just maintain bindings for all of the cpp libraries,
>> > > > rather than of having a hybrid solution.
>> > > >
>> > > > While We could spare the reimplementation of gandiva, it'd make
>> > > > packaging more complicated and rust development way less
>> > > > welcoming to new contributors.
>> > > >
>> > > > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Now that the Rust implementation of Arrow is maturing, I'm
>> interested
>> > > in
>> > > > > having bindings for Gandiva for query execution, rather than
>> > > duplicating
>> > > > > this in Rust.
>> > > > >
>> > > > > I will likely start looking at this soon but wanted to see if
>> anyone
>> > > else
>> > > > > here is particularly interested in this area of functionality?
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Andy.
>> > > > >
>> > >
>> >
>>
>
>
> --
> Renjie Liu
> Software Engineer, MVAD
>
Re: Rust bindings for Gandiva
Posted by Renjie Liu <li...@gmail.com>.
Hi:
@Andy Grove <an...@gmail.com> Are you developing this? I'm interested
in this and want to join development.
On Tue, Jan 8, 2019 at 3:18 PM Praveen Kumar <pr...@dremio.com> wrote:
> Agree with Wes, the protobuf based interface should be the language neutral
> way to build expressions with Gandiva.
>
> On Mon, Jan 7, 2019 at 8:30 PM Andy Grove <an...@gmail.com> wrote:
>
> > This makes sense to me know that I understand a little more about
> Gandiva.
> > This also fits well with my proposal to donate DataFusion in the other
> > thread. DataFusion can manage the overall logical query plan in Rust and
> > potentially delegate some subset of expression evaluation to Gandiva via
> > protobuf.
> >
> > Thanks,
> >
> > Andy.
> >
> > On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > Gandiva supports a Protobuf-based interface -- this is how Java
> > > interacts with it via JNI. Rust could do the same -- that would
> > > probably be easier than wrapping the C++ class structure. It would
> > > also help drive new feature requirements in the serialized
> > > projection/filter expression trees
> > >
> > > - Wes
> > >
> > > On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
> > > <sz...@gmail.com> wrote:
> > > >
> > > > I'm not sure, that a binding is a good idea. Both Arrow and Parquet
> > > > already have their own rust implementation, and a interfacing with
> > > > cpp isn't as easy and straightforward than it is with C. Otherwise
> > > > We could simply just maintain bindings for all of the cpp libraries,
> > > > rather than of having a hybrid solution.
> > > >
> > > > While We could spare the reimplementation of gandiva, it'd make
> > > > packaging more complicated and rust development way less
> > > > welcoming to new contributors.
> > > >
> > > > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com>
> > wrote:
> > > >
> > > > > Now that the Rust implementation of Arrow is maturing, I'm
> interested
> > > in
> > > > > having bindings for Gandiva for query execution, rather than
> > > duplicating
> > > > > this in Rust.
> > > > >
> > > > > I will likely start looking at this soon but wanted to see if
> anyone
> > > else
> > > > > here is particularly interested in this area of functionality?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Andy.
> > > > >
> > >
> >
>
--
Renjie Liu
Software Engineer, MVAD
Re: Rust bindings for Gandiva
Posted by Praveen Kumar <pr...@dremio.com>.
Agree with Wes, the protobuf based interface should be the language neutral
way to build expressions with Gandiva.
On Mon, Jan 7, 2019 at 8:30 PM Andy Grove <an...@gmail.com> wrote:
> This makes sense to me know that I understand a little more about Gandiva.
> This also fits well with my proposal to donate DataFusion in the other
> thread. DataFusion can manage the overall logical query plan in Rust and
> potentially delegate some subset of expression evaluation to Gandiva via
> protobuf.
>
> Thanks,
>
> Andy.
>
> On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <we...@gmail.com> wrote:
>
> > Gandiva supports a Protobuf-based interface -- this is how Java
> > interacts with it via JNI. Rust could do the same -- that would
> > probably be easier than wrapping the C++ class structure. It would
> > also help drive new feature requirements in the serialized
> > projection/filter expression trees
> >
> > - Wes
> >
> > On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
> > <sz...@gmail.com> wrote:
> > >
> > > I'm not sure, that a binding is a good idea. Both Arrow and Parquet
> > > already have their own rust implementation, and a interfacing with
> > > cpp isn't as easy and straightforward than it is with C. Otherwise
> > > We could simply just maintain bindings for all of the cpp libraries,
> > > rather than of having a hybrid solution.
> > >
> > > While We could spare the reimplementation of gandiva, it'd make
> > > packaging more complicated and rust development way less
> > > welcoming to new contributors.
> > >
> > > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > > Now that the Rust implementation of Arrow is maturing, I'm interested
> > in
> > > > having bindings for Gandiva for query execution, rather than
> > duplicating
> > > > this in Rust.
> > > >
> > > > I will likely start looking at this soon but wanted to see if anyone
> > else
> > > > here is particularly interested in this area of functionality?
> > > >
> > > > Thanks,
> > > >
> > > > Andy.
> > > >
> >
>
Re: Rust bindings for Gandiva
Posted by Andy Grove <an...@gmail.com>.
This makes sense to me know that I understand a little more about Gandiva.
This also fits well with my proposal to donate DataFusion in the other
thread. DataFusion can manage the overall logical query plan in Rust and
potentially delegate some subset of expression evaluation to Gandiva via
protobuf.
Thanks,
Andy.
On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <we...@gmail.com> wrote:
> Gandiva supports a Protobuf-based interface -- this is how Java
> interacts with it via JNI. Rust could do the same -- that would
> probably be easier than wrapping the C++ class structure. It would
> also help drive new feature requirements in the serialized
> projection/filter expression trees
>
> - Wes
>
> On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > I'm not sure, that a binding is a good idea. Both Arrow and Parquet
> > already have their own rust implementation, and a interfacing with
> > cpp isn't as easy and straightforward than it is with C. Otherwise
> > We could simply just maintain bindings for all of the cpp libraries,
> > rather than of having a hybrid solution.
> >
> > While We could spare the reimplementation of gandiva, it'd make
> > packaging more complicated and rust development way less
> > welcoming to new contributors.
> >
> > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com> wrote:
> >
> > > Now that the Rust implementation of Arrow is maturing, I'm interested
> in
> > > having bindings for Gandiva for query execution, rather than
> duplicating
> > > this in Rust.
> > >
> > > I will likely start looking at this soon but wanted to see if anyone
> else
> > > here is particularly interested in this area of functionality?
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
>
Re: Rust bindings for Gandiva
Posted by Wes McKinney <we...@gmail.com>.
Gandiva supports a Protobuf-based interface -- this is how Java
interacts with it via JNI. Rust could do the same -- that would
probably be easier than wrapping the C++ class structure. It would
also help drive new feature requirements in the serialized
projection/filter expression trees
- Wes
On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> I'm not sure, that a binding is a good idea. Both Arrow and Parquet
> already have their own rust implementation, and a interfacing with
> cpp isn't as easy and straightforward than it is with C. Otherwise
> We could simply just maintain bindings for all of the cpp libraries,
> rather than of having a hybrid solution.
>
> While We could spare the reimplementation of gandiva, it'd make
> packaging more complicated and rust development way less
> welcoming to new contributors.
>
> On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com> wrote:
>
> > Now that the Rust implementation of Arrow is maturing, I'm interested in
> > having bindings for Gandiva for query execution, rather than duplicating
> > this in Rust.
> >
> > I will likely start looking at this soon but wanted to see if anyone else
> > here is particularly interested in this area of functionality?
> >
> > Thanks,
> >
> > Andy.
> >
Re: Rust bindings for Gandiva
Posted by Krisztián Szűcs <sz...@gmail.com>.
I'm not sure, that a binding is a good idea. Both Arrow and Parquet
already have their own rust implementation, and a interfacing with
cpp isn't as easy and straightforward than it is with C. Otherwise
We could simply just maintain bindings for all of the cpp libraries,
rather than of having a hybrid solution.
While We could spare the reimplementation of gandiva, it'd make
packaging more complicated and rust development way less
welcoming to new contributors.
On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <an...@gmail.com> wrote:
> Now that the Rust implementation of Arrow is maturing, I'm interested in
> having bindings for Gandiva for query execution, rather than duplicating
> this in Rust.
>
> I will likely start looking at this soon but wanted to see if anyone else
> here is particularly interested in this area of functionality?
>
> Thanks,
>
> Andy.
>