Fwd: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

[Julian Hyde <jh...@gmail.com>]

On dev@arrow, Wes McKinney just started a discussion of a language for “programming” Arrow. One of its explicit goals is for systems like Calcite to generate such code as their physical plan.

This would be great for Calcite. Arrow started as an excellent runtime for high-performance data processing.

Julian


> Begin forwarded message:
> 
> From: Wes McKinney <we...@gmail.com>
> Subject: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines
> Date: August 2, 2021 at 5:16:50 PM PDT
> To: dev <de...@arrow.apache.org>
> Reply-To: dev@arrow.apache.org
> 
> hi folks,
> 
> This idea came up in passing in the past -- given that there are
> multiple independent efforts to develop Arrow-native query engines
> (and surely many more to come), it seems like it would be valuable to
> have a way to enable user languages (like Java, Python, R, or Rust,
> for example) to communicate with backend computing engines (like
> DataFusion, or new computing capabilities being built in the Arrow C++
> library) in a fashion that is "lower-level" than SQL and specialized
> to Arrow's type system. So rather than leaving it to a SQL parser /
> analyzer framework to generate an expression tree of relational
> operators and then translate that to an Arrow-native compute-engine's
> internal grammer, a user framework could provide the desired
> Arrow-native expression tree / data manipulations directly and skip
> the SQL altogether.
> 
> The idea of creating a "serialized intermediate representation (IR)"
> for Arrow compute operations would be to serve use cases large and
> small -- from the most complex TPC-* or time series database query to
> the most simple array predicate/filter sent with an RPC request using
> Arrow Flight. It is deliberately language- and engine-agnostic, with
> the only commonality across usages being the Arrow columnar format
> (schemas and array types). This would be better than leaving it to
> each application to develop its own bespoke expression representations
> for its needs.
> 
> I spent a while thinking about this and wrote up a brain dump RFC
> document [1] and accompanying pull request [2] that makes the possibly
> controversial choice of using Flatbuffers to represent the serialized
> IR. I discuss the rationale for the choice of Flatbuffers in the RFC
> document. This PR is obviously deficient in many regards (incomplete,
> hacky, or unclear in places), and will need some help from others to
> flesh out. I suspect that actually implementing the IR will be
> necessary to work out many of the low-level details.
> 
> Note that this IR is intended to be more of a "superset" project than
> a "lowest common denominator". So there may be things that it includes
> which are only available in some engines (e.g. engines that have
> specialized handling of time series data).
> 
> As some of my personal motivation for the project, concurrent with the
> genesis of Apache Arrow, I started a Python project called Ibis [3]
> (which is similar to R's dplyr project) which serves as a "Python
> analytical query IR builder" that is capable of generating most of the
> SQL standard, targeting many different SQL dialects and other backends
> (like pandas). Microsoft ([4]) and Google ([5]) have used this library
> as a "many-SQL" middleware. As such, I would like to be able to
> translate between the in-memory "logical query" data structures in a
> library like Ibis to a serialized format that can be executed by many
> different Arrow-native query engines. The expression primitives
> available in Ibis should serve as helpful test cases, too.
> 
> I look forward to the community's comments on the RFC document [1] and
> pull request [2] -- I realize that arriving at consensus on a complex
> and ambitious project like this can be challenging so I recommend
> spending time on the "non-goals" section in the RFC and ask questions
> if you are unclear about the scope of what problems this is trying to
> solve. I would be happy to give Edit access on the RFC document to
> others and would consider ideas about how to move forward with
> something that is able to be implemented by different Arrow libraries
> in the reasonably near future.
> 
> Thanks,
> Wes
> 
> [1]: https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit#
> [2]: https://github.com/apache/arrow/pull/10856
> [3]: https://ibis-project.org/
> [4]: http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf
> [5]: https://cloud.google.com/blog/products/databases/automate-data-validation-with-dvt