RE: [ext] Re: language independent representation of filter expressions

[Patrick Pai <pp...@DRWHoldings.com>]

I only briefly looked into the Gandiva protobuf, but one issue seems to be using protobuf (Wes is against this for dependency reasons). There's also some inconsistencies between the Gandiva protobuf and how filter expressions should be represented, i.e. in the Gandiva protobuf fields are typed when I think fields should just contain a field name.

-----Original Message-----
From: Jacques Nadeau <ja...@apache.org>
Sent: Thursday, July 23, 2020 10:14 PM
To: dev <de...@arrow.apache.org>
Subject: [ext] Re: language independent representation of filter expressions

Have you tried to use the existing expression representation provided by Gandiva? What are the issues you've seen with it?

On Wed, Jul 22, 2020 at 10:24 AM Patrick Pai <pp...@drwholdings.com> wrote:

> Hi all,
>
> After some discussion with Steve, we'd like to propose and get
> feedback on an alternative to representing expressions entirely with flatbuffers.
>
> To give some context, we thought about how we'd construct flatbuffer
> expressions in Java or another language if we went down that route. We
> realized that it'd be possible, but not user friendly. An example is
> specifying an array of int values in Java for an InExpression. In
> Java, we'd ideally have some user-friendly class (i.e. arrow's
> IntVector) that then gets converted to the appropriate flatbuffer
> representation. I think this is what Jacques was saying about language
> support being too weak - it's possible for Java users to construct a
> flatbuffer expression, but not easily without an additional conversion layer for every language.
>
> An alternative we're thinking about is to only represent enum values (i.e.
> those defined in arrow::dataset::ExpressionType::type) in a flatbuffer
> schema, and rely on the existing IPC format (used to
> serialize/deserialize cpp expressions) to pass the struct array
> representation of an expression from for example Java to C++. The one
> difference is in the struct array representation, we use the enum
> values defined in our flatbuffer schema instead of existing cpp enums.
> This approach requires us on the Java side (and languages other than
> C++) to construct the struct array, but the benefit is minimal changes
> to the C++ code (the main change being using our flatbuffer schema enum values).
>
>
> On 2020/07/13 09:21:19, Antoine Pitrou <so...@pitrou.net> wrote:
> > On Sat, 11 Jul 2020 09:55:16 -0700
> > Jacques Nadeau <ja...@apache.org> wrote:
> > >
> > > I'm against extending use of flatbuf within Arrow. The language
> support is
> > > too weak. Language support isn't just about having a binding for
> different
> > > languages, it is about having a high-quality binding.
> >
> > Could you please expand on this?  ("the language support is too
> > weak")
> >
> > Thank you
> >
> > Antoine.
> >
> >
> >
>
> This e-mail and any attachments may contain information that is
> confidential and proprietary and otherwise protected from disclosure.
> If you are not the intended recipient of this e-mail, do not read,
> duplicate or redistribute it by any means. Please immediately delete
> it and any attachments and notify the sender that you have received it by mistake.
> Unintended recipients are prohibited from taking action on the basis
> of information in this e-mail or any attachments. The DRW Companies
> make no representations that this e-mail or any attachments are free
> of computer viruses or other defects.
>
This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

Re: [ext] Re: language independent representation of filter expressions

[Wes McKinney <we...@gmail.com>]

I am OK with using the .proto files for now while the serialization
protocol is in development and focusing on capturing the functional
requirements and leaving the Protobuf vs Flatbuffers debate for later.

I don't think that JSON is an adequate substitute, because if Protobuf
is the desired / official serialization protocol, then we must ship
C++ libraries that support it, which means pulling in the
libprotobuf.a dependency. This also creates a transitive dependency on
Protocol Buffers for any third party library that wants to import or
export the serialized expressions, so it won't just be the Arrow C++
libraries that are impacted, but also any library that wants to
participate in the ecosystem. Flatbuffers by contrast can be used in
C/C++ applications without introducing any build or runtime
dependencies (indeed, when you do "cmake .; make" in the Arrow C++
project now, Flatbuffers is not required to have IPC protocol support
because we have the Flatbuffers header files and generated sources
checked in to the repo). This is a meaningful tax on the ecosystem
that IMHO significantly outweighs the negative effects of a worse API
/ developer UX when using Flatbuffers in Java.

On Fri, Jul 24, 2020 at 3:28 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> Per my other email, you can generate JSON that is canonical protobuf if you
> don't want to pull the protobuf dependency. In terms of field typing: I
> could see treating that as optional in a user expression that is resolved
> later.
>
> On Fri, Jul 24, 2020 at 12:47 PM Patrick Pai <pp...@drwholdings.com> wrote:
>
> > I only briefly looked into the Gandiva protobuf, but one issue seems to be
> > using protobuf (Wes is against this for dependency reasons). There's also
> > some inconsistencies between the Gandiva protobuf and how filter
> > expressions should be represented, i.e. in the Gandiva protobuf fields are
> > typed when I think fields should just contain a field name.
> >
> > -----Original Message-----
> > From: Jacques Nadeau <ja...@apache.org>
> > Sent: Thursday, July 23, 2020 10:14 PM
> > To: dev <de...@arrow.apache.org>
> > Subject: [ext] Re: language independent representation of filter
> > expressions
> >
> > Have you tried to use the existing expression representation provided by
> > Gandiva? What are the issues you've seen with it?
> >
> > On Wed, Jul 22, 2020 at 10:24 AM Patrick Pai <pp...@drwholdings.com> wrote:
> >
> > > Hi all,
> > >
> > > After some discussion with Steve, we'd like to propose and get
> > > feedback on an alternative to representing expressions entirely with
> > flatbuffers.
> > >
> > > To give some context, we thought about how we'd construct flatbuffer
> > > expressions in Java or another language if we went down that route. We
> > > realized that it'd be possible, but not user friendly. An example is
> > > specifying an array of int values in Java for an InExpression. In
> > > Java, we'd ideally have some user-friendly class (i.e. arrow's
> > > IntVector) that then gets converted to the appropriate flatbuffer
> > > representation. I think this is what Jacques was saying about language
> > > support being too weak - it's possible for Java users to construct a
> > > flatbuffer expression, but not easily without an additional conversion
> > layer for every language.
> > >
> > > An alternative we're thinking about is to only represent enum values
> > (i.e.
> > > those defined in arrow::dataset::ExpressionType::type) in a flatbuffer
> > > schema, and rely on the existing IPC format (used to
> > > serialize/deserialize cpp expressions) to pass the struct array
> > > representation of an expression from for example Java to C++. The one
> > > difference is in the struct array representation, we use the enum
> > > values defined in our flatbuffer schema instead of existing cpp enums.
> > > This approach requires us on the Java side (and languages other than
> > > C++) to construct the struct array, but the benefit is minimal changes
> > > to the C++ code (the main change being using our flatbuffer schema enum
> > values).
> > >
> > >
> > > On 2020/07/13 09:21:19, Antoine Pitrou <so...@pitrou.net> wrote:
> > > > On Sat, 11 Jul 2020 09:55:16 -0700
> > > > Jacques Nadeau <ja...@apache.org> wrote:
> > > > >
> > > > > I'm against extending use of flatbuf within Arrow. The language
> > > support is
> > > > > too weak. Language support isn't just about having a binding for
> > > different
> > > > > languages, it is about having a high-quality binding.
> > > >
> > > > Could you please expand on this?  ("the language support is too
> > > > weak")
> > > >
> > > > Thank you
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > >
> > > This e-mail and any attachments may contain information that is
> > > confidential and proprietary and otherwise protected from disclosure.
> > > If you are not the intended recipient of this e-mail, do not read,
> > > duplicate or redistribute it by any means. Please immediately delete
> > > it and any attachments and notify the sender that you have received it
> > by mistake.
> > > Unintended recipients are prohibited from taking action on the basis
> > > of information in this e-mail or any attachments. The DRW Companies
> > > make no representations that this e-mail or any attachments are free
> > > of computer viruses or other defects.
> > >
> > This e-mail and any attachments may contain information that is
> > confidential and proprietary and otherwise protected from disclosure. If
> > you are not the intended recipient of this e-mail, do not read, duplicate
> > or redistribute it by any means. Please immediately delete it and any
> > attachments and notify the sender that you have received it by mistake.
> > Unintended recipients are prohibited from taking action on the basis of
> > information in this e-mail or any attachments. The DRW Companies make no
> > representations that this e-mail or any attachments are free of computer
> > viruses or other defects.
> >

Re: [ext] Re: language independent representation of filter expressions

[Jacques Nadeau <ja...@apache.org>]

Per my other email, you can generate JSON that is canonical protobuf if you
don't want to pull the protobuf dependency. In terms of field typing: I
could see treating that as optional in a user expression that is resolved
later.

On Fri, Jul 24, 2020 at 12:47 PM Patrick Pai <pp...@drwholdings.com> wrote:

> I only briefly looked into the Gandiva protobuf, but one issue seems to be
> using protobuf (Wes is against this for dependency reasons). There's also
> some inconsistencies between the Gandiva protobuf and how filter
> expressions should be represented, i.e. in the Gandiva protobuf fields are
> typed when I think fields should just contain a field name.
>
> -----Original Message-----
> From: Jacques Nadeau <ja...@apache.org>
> Sent: Thursday, July 23, 2020 10:14 PM
> To: dev <de...@arrow.apache.org>
> Subject: [ext] Re: language independent representation of filter
> expressions
>
> Have you tried to use the existing expression representation provided by
> Gandiva? What are the issues you've seen with it?
>
> On Wed, Jul 22, 2020 at 10:24 AM Patrick Pai <pp...@drwholdings.com> wrote:
>
> > Hi all,
> >
> > After some discussion with Steve, we'd like to propose and get
> > feedback on an alternative to representing expressions entirely with
> flatbuffers.
> >
> > To give some context, we thought about how we'd construct flatbuffer
> > expressions in Java or another language if we went down that route. We
> > realized that it'd be possible, but not user friendly. An example is
> > specifying an array of int values in Java for an InExpression. In
> > Java, we'd ideally have some user-friendly class (i.e. arrow's
> > IntVector) that then gets converted to the appropriate flatbuffer
> > representation. I think this is what Jacques was saying about language
> > support being too weak - it's possible for Java users to construct a
> > flatbuffer expression, but not easily without an additional conversion
> layer for every language.
> >
> > An alternative we're thinking about is to only represent enum values
> (i.e.
> > those defined in arrow::dataset::ExpressionType::type) in a flatbuffer
> > schema, and rely on the existing IPC format (used to
> > serialize/deserialize cpp expressions) to pass the struct array
> > representation of an expression from for example Java to C++. The one
> > difference is in the struct array representation, we use the enum
> > values defined in our flatbuffer schema instead of existing cpp enums.
> > This approach requires us on the Java side (and languages other than
> > C++) to construct the struct array, but the benefit is minimal changes
> > to the C++ code (the main change being using our flatbuffer schema enum
> values).
> >
> >
> > On 2020/07/13 09:21:19, Antoine Pitrou <so...@pitrou.net> wrote:
> > > On Sat, 11 Jul 2020 09:55:16 -0700
> > > Jacques Nadeau <ja...@apache.org> wrote:
> > > >
> > > > I'm against extending use of flatbuf within Arrow. The language
> > support is
> > > > too weak. Language support isn't just about having a binding for
> > different
> > > > languages, it is about having a high-quality binding.
> > >
> > > Could you please expand on this?  ("the language support is too
> > > weak")
> > >
> > > Thank you
> > >
> > > Antoine.
> > >
> > >
> > >
> >
> > This e-mail and any attachments may contain information that is
> > confidential and proprietary and otherwise protected from disclosure.
> > If you are not the intended recipient of this e-mail, do not read,
> > duplicate or redistribute it by any means. Please immediately delete
> > it and any attachments and notify the sender that you have received it
> by mistake.
> > Unintended recipients are prohibited from taking action on the basis
> > of information in this e-mail or any attachments. The DRW Companies
> > make no representations that this e-mail or any attachments are free
> > of computer viruses or other defects.
> >
> This e-mail and any attachments may contain information that is
> confidential and proprietary and otherwise protected from disclosure. If
> you are not the intended recipient of this e-mail, do not read, duplicate
> or redistribute it by any means. Please immediately delete it and any
> attachments and notify the sender that you have received it by mistake.
> Unintended recipients are prohibited from taking action on the basis of
> information in this e-mail or any attachments. The DRW Companies make no
> representations that this e-mail or any attachments are free of computer
> viruses or other defects.
>