[DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Hello,

One thing that was discussed in the sync call is the ability to easily
pass arrays at runtime between Arrow implementations or Arrow-supporting
libraries in the same process, without bearing the cost of linking to
e.g. the C++ Arrow library.

(for example: "Duckdb wants to provide an option to return Arrow data of
result sets, but they don't like having Arrow as a dependency")

One possibility would be to define a C-level protocol similar in spirit
to the Python buffer protocol, which some people may be familiar with (*).

The basic idea is to define a simple C struct, which is ABI-stable and
describes an Arrow away adequately.  The struct can be stack-allocated.
Its definition can also be copied in another project (or interfaced with
using a C FFI layer, depending on the language).

There is no formal proposal, this message is meant to stir the discussion.

Issues to work out:

* Memory lifetime issues: where Python simply associates the Py_buffer
with a PyObject owner (a garbage-collected Python object), we need
another means to control lifetime of pointed areas.  One simple
possibility is to include a destructor function pointer in the protocol
struct.

* Arrow type representation.  We probably need some kind of "format"
mini-language to represent Arrow types, so that a type can be described
using a `const char*`.  Ideally, primitives types at least should be
trivially parsable.  We may take inspiration from Python here (`struct`
module format characters, PEP 3118 format additions).

Example C struct definition (not a formal proposal!):

struct ArrowBuffer {
  void* data;
  int64_t nbytes;
  // Called by the consumer when it doesn't need the buffer anymore
  void (*release)(struct ArrowBuffer*);
  // Opaque user data (for e.g. the release callback)
  void* user_data;
};

struct ArrowArray {
  // Type description
  const char* format;
  // Data description
  int64_t length;
  int64_t null_count;
  int64_t n_buffers;
  // Note: this pointers are probably owned by the ArrowArray struct
  // and will be released and free()ed by the release callback.
  struct BufferDescriptor* buffers;
  struct ArrowDescriptor* dictionary;
  // Called by the consumer when it doesn't need the array anymore
  void (*release)(struct ArrowArrayDescriptor*);
  // Opaque user data (for e.g. the release callback)
  void* user_data;
};

Thoughts?

(*) For the record, the reference for the Python buffer protocol:
https://docs.python.org/3/c-api/buffer.html#buffer-structure
and its C struct definition:
https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195

Regards

Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 29/09/2019 à 19:59, Jacques Nadeau a écrit :
> 
> It seems like you're saying: "flatbuffers is too complex an encoding, let's
> create a new encoding".

Most of the spec is a plain C-level struct in the native ABI, so it
avoids any kind of encoding issue.  And, yes, flatbuffers must be dealt
with and it's not something most people are used to.

> You're talking about three separate things:
> 1) How do I communicate schema
> 2) How do I communicate encoded batches with zero copy
> 3) How do I define rules around ownership semantics
[...]

> I agree for #1 but don't agree for #2

What exactly don't you agree with?

>
 and think we
> should do our best to avoid an alternative for #2. We don't have something
> that defines #3 so that is clearly a gap. The definition should be done
> with all languages in mind.

Well, is there anything that doesn't work well with Java here?  It would
be good to know, because you don't seem to be attacking any specific
aspect of the spec.  The spec is designed to work well with any language
where a decent C FFI is available.

> To introduce this, I also think we need to have reference
> tools/implementation/integration tests for both C++ and Java.

I will definitely do the C++ part.  I'm afraid I'm not qualified to do
the Java part, though.

Regards

Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

On Sun, Sep 29, 2019 at 12:59 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
> > * No dependency on Flatbuffers.
> > * No buffer reassembly (data is already exposed in logical Arrow format).
> > * Zero-copy by design.
> > * Easy to reimplement from scratch.
> >
> > I don't see how the flatbuffer pattern for data headers doesn't
> accomplish
> > all of these things. At its definition, is a very simple representation
> of
> > data that could be worked with independently of the flatbuffers codebase.
> > It was designed so systems could map directly into that memory without
> > interacting with a flatbuffers library.
> >
> > Specifically the following three structures were designed to already
> allow
> > what I think this proposal is trying to recreate. All three are very
> simple
> > to construct in a direct, non-flatbuffer dependent read/write pattern.
>
> Are they?  Personally, I wouldn't know how to do that.  I don't know
> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
> it be? if it's portable accross different platforms, then it's probably
> not compatible with any particular platform's C ABI, or only as a
> conincidence), how I'm supposed to make use of the "offset" field, or
> what the lifetime / ownership of all this data is.
>
> I may be missing something, but if the answer is that it's easy to
> reimplement Flatbuffers' encoding without relying on the Flatbuffers
> project's source code, I'm a bit skeptical.
>
> Regards
>
> Antoine.
>


You're talking about three separate things:
1) How do I communicate schema
2) How do I communicate encoded batches with zero copy
3) How do I define rules around ownership semantics

Flatbuffers isn't trying to solve #3 but it is definitely trying to solve 1
& 2. Flatbuffers is a formal in memory specification of how data is layed
out, just like Arrow. While some constructs or more complex to construct,
the record batch construct was designed specifically to avoid those
patterns. That's why we only use structs within that construct.

"...it's easy to reimplement Flatbuffers' encoding...I'm a bit skeptical"

Let's evaluate that path before dismissing it.
https://google.github.io/flatbuffers/md__internals.html

If I remember correctly, the structure of a flatbuffers record batch is:

[int8-]
[uint4-number of field nodes]
  [int8-length][int8-null count]
  [int8-length][int8-null count]
  ...
[uint4-number of buffers]
  [int8-memory address][int8-length of buffer]
  [int8-memory address][int8-length of buffer]
  ...

The goal of offset was offset in the memory space you are within. In a
single memory space, that is equivalent to memory address. Given a known
schema, you have a known address for all other accesses across all batches
without any specialized reading.

It seems like you're saying: "flatbuffers is too complex an encoding, let's
create a new encoding". I agree for #1 but don't agree for #2 and think we
should do our best to avoid an alternative for #2. We don't have something
that defines #3 so that is clearly a gap. The definition should be done
with all languages in mind. I also am generally against implementing a new
terse encoding method and specialized parsing.

To introduce this, I also think we need to have reference
tools/implementation/integration tests for both C++ and Java.

Re: [DISCUSS] raw pointers and FFI (C-level in-process array protocol)

[Sutou Kouhei <ko...@clear-code.com>]

Hi,

I think that FFI use is misleading. Normally, language
bindings for this API are useless for processing Apache
Arrow data. Because these bindings of this API can only
import/export Apache Arrow data. Target language may not
have useful/fast API for processing imported Apache Arrow
data. For example, Julia may process imported Apache Arrow
data with Julia's built-in feature. Other script
languages may not, even LuaJIT.

We need multiple languages in one process for in-process
use. There are some approaches for this situation. Actually
some approaches are used but these approaches are minor. (I
think.)


I think that interacting to Apache Arrow ready library is a
useful use case of this API.

If SQLite uses this API to return result set in Apache Arrow
format, it'll be useful. SQLite doesn't need additional
dependency to add support for exporting in Apache Arrow
format. SQLite will return schema by its existing API such
as sqlite3_column_type() and return data with this API.
SQLite bindings can add Apache Arrow data export API easily
because it's just raw C API. (FFI may be used to bind the
Apache Arrow data export API.)

SQLite doesn't need to process Apache Arrow data. It just
exports Apache Arrow data. So this API is enough.


This API will be useful for libraries that want to support
just Apache Arrow data import/export.


Thanks,
--
kou

In <CA...@mail.gmail.com>
  "Re: [DISCUSS] raw pointers and FFI (C-level in-process array protocol)" on Thu, 3 Oct 2019 11:17:29 -0500,
  Wes McKinney <we...@gmail.com> wrote:

> Related: Gandiva invented its own particular way of passing memory
> addresses through the JNI boundary rather than using Flatbuffers
> messages
> 
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/jni/jni_common.cc#L505
> 
> I'm all for language-agnostic in-memory data passing, but there is a
> use case for a C API to pass pointers at call sites while avoiding
> flattening (disassembly) and unflattening (reassembly) steps.
> 
> On Thu, Oct 3, 2019 at 4:34 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Hi Jacques,
>>
>> Le 03/10/2019 à 02:46, Jacques Nadeau a écrit :
>> >
>> > I think it is reasonable to argue that keeping any ABI (or header/struct
>> > pattern) as narrow as possible would allow us to minimize overlap with the
>> > existing in-memory specification. In Arrow's case, this could be as simple
>> > as a single memory pointer for schema (backed by flatbuffers) and a single
>> > memory location for data (that references the record batch header, which in
>> > turn provides pointers into the actual arrow data). [...]
>> >
>> > [...] (For example, in a JVM
>> > view of the world, working with a plain struct in java rather than a set of
>> > memory pointers against our existing IPC formats would be quite painful and
>> > we'd definitely need to create some glue code for users. I worry the same
>> > pattern would occur in many other languages.)
>>
>> I'm trying to understand the point you're making.  Here you say that it
>> was difficult for the JVM to deal with raw pointers.  But above you seem
>> to argue for a flatbuffers-based serialization containing raw pointers.
>>
>> Here's another way to frame the question: how do you propose to do
>> zero-copy between different languages if not by passing raw pointers to
>> the Arrow data?  And if passing raw pointers is acceptable, what is
>> wrong with the spec as proposed?
>>
>>
>> As for creating glue code: yes, of course, that would be needed in most
>> languages that want to provide this interface (including C++).  You do
>> need a C FFI for that.  I'm quite sure it would be possible to implement
>> this proposal in pure Python with ctypes / cffi, for example (as a toy
>> example, since PyArrow exists :-)).  When writing the spec, I also took
>> a look at the Go and Rust FFIs, and they seem good enough to interact
>> with it.  I tried to take a look at JNI, but of course I got lost in the
>> documentation :-)
>>
>> If you are worried that people start thinking that this proposal is part
>> of the Arrow specification, perhaps we can make it clear that exposing
>> this interface is optional for implementations.
>>
>> Regards
>>
>> Antoine.

Re: [DISCUSS] raw pointers and FFI (C-level in-process array protocol)

[Wes McKinney <we...@gmail.com>]

Related: Gandiva invented its own particular way of passing memory
addresses through the JNI boundary rather than using Flatbuffers
messages

https://github.com/apache/arrow/blob/master/cpp/src/gandiva/jni/jni_common.cc#L505

I'm all for language-agnostic in-memory data passing, but there is a
use case for a C API to pass pointers at call sites while avoiding
flattening (disassembly) and unflattening (reassembly) steps.

On Thu, Oct 3, 2019 at 4:34 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Jacques,
>
> Le 03/10/2019 à 02:46, Jacques Nadeau a écrit :
> >
> > I think it is reasonable to argue that keeping any ABI (or header/struct
> > pattern) as narrow as possible would allow us to minimize overlap with the
> > existing in-memory specification. In Arrow's case, this could be as simple
> > as a single memory pointer for schema (backed by flatbuffers) and a single
> > memory location for data (that references the record batch header, which in
> > turn provides pointers into the actual arrow data). [...]
> >
> > [...] (For example, in a JVM
> > view of the world, working with a plain struct in java rather than a set of
> > memory pointers against our existing IPC formats would be quite painful and
> > we'd definitely need to create some glue code for users. I worry the same
> > pattern would occur in many other languages.)
>
> I'm trying to understand the point you're making.  Here you say that it
> was difficult for the JVM to deal with raw pointers.  But above you seem
> to argue for a flatbuffers-based serialization containing raw pointers.
>
> Here's another way to frame the question: how do you propose to do
> zero-copy between different languages if not by passing raw pointers to
> the Arrow data?  And if passing raw pointers is acceptable, what is
> wrong with the spec as proposed?
>
>
> As for creating glue code: yes, of course, that would be needed in most
> languages that want to provide this interface (including C++).  You do
> need a C FFI for that.  I'm quite sure it would be possible to implement
> this proposal in pure Python with ctypes / cffi, for example (as a toy
> example, since PyArrow exists :-)).  When writing the spec, I also took
> a look at the Go and Rust FFIs, and they seem good enough to interact
> with it.  I tried to take a look at JNI, but of course I got lost in the
> documentation :-)
>
> If you are worried that people start thinking that this proposal is part
> of the Arrow specification, perhaps we can make it clear that exposing
> this interface is optional for implementations.
>
> Regards
>
> Antoine.

Re: [DISCUSS] raw pointers and FFI (C-level in-process array protocol)

[Antoine Pitrou <an...@python.org>]

Hi Jacques,

Le 03/10/2019 à 02:46, Jacques Nadeau a écrit :
> 
> I think it is reasonable to argue that keeping any ABI (or header/struct
> pattern) as narrow as possible would allow us to minimize overlap with the
> existing in-memory specification. In Arrow's case, this could be as simple
> as a single memory pointer for schema (backed by flatbuffers) and a single
> memory location for data (that references the record batch header, which in
> turn provides pointers into the actual arrow data). [...]
> 
> [...] (For example, in a JVM
> view of the world, working with a plain struct in java rather than a set of
> memory pointers against our existing IPC formats would be quite painful and
> we'd definitely need to create some glue code for users. I worry the same
> pattern would occur in many other languages.)

I'm trying to understand the point you're making.  Here you say that it
was difficult for the JVM to deal with raw pointers.  But above you seem
to argue for a flatbuffers-based serialization containing raw pointers.

Here's another way to frame the question: how do you propose to do
zero-copy between different languages if not by passing raw pointers to
the Arrow data?  And if passing raw pointers is acceptable, what is
wrong with the spec as proposed?


As for creating glue code: yes, of course, that would be needed in most
languages that want to provide this interface (including C++).  You do
need a C FFI for that.  I'm quite sure it would be possible to implement
this proposal in pure Python with ctypes / cffi, for example (as a toy
example, since PyArrow exists :-)).  When writing the spec, I also took
a look at the Go and Rust FFIs, and they seem good enough to interact
with it.  I tried to take a look at JNI, but of course I got lost in the
documentation :-)

If you are worried that people start thinking that this proposal is part
of the Arrow specification, perhaps we can make it clear that exposing
this interface is optional for implementations.

Regards

Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

On Wed, Oct 2, 2019 at 10:19 PM Wes McKinney <we...@gmail.com> wrote:
>
> On Wed, Oct 2, 2019 at 7:46 PM Jacques Nadeau <ja...@apache.org> wrote:
> >
> > I'd like to hear more opinions from others on this topic. This conversation
> > seems mostly dominated by comments from myself, Wes and Antoine.
> >
> > I think it is reasonable to argue that keeping any ABI (or header/struct
> > pattern) as narrow as possible would allow us to minimize overlap with the
> > existing in-memory specification. In Arrow's case, this could be as simple
> > as a single memory pointer for schema (backed by flatbuffers) and a single
> > memory location for data (that references the record batch header, which in
> > turn provides pointers into the actual arrow data). Extensions would need
> > to be added for reference management as done here but I continue to think
> > we should defer discussion of that until the base data structures are
> > resolved. I see the comments here as arguing for a much broader ABI, in
> > part to support having people build "Arrow" components that interconnect
> > using this new interface. I understand the desire to expand the ABI to be
> > driven by needs to reduce dependencies and ease usability.
> >
> > The representation within the related patch is being presented as a way for
> > applications to share Arrow data but is not easily accessible to all
> > languages. I want to avoid a situation where someone says "I produced an
> > Arrow API" when what they've really done is created a C interface which
> > only a small subset of languages can actually leverage. For example, every
> > language now knows how to parse the existing schema definition as rendered
> > in flatbuf. In order to interact with something that implements this new
> > pattern one would also be required to implement completely new schema
> > consumption code. In the proposal itself it suggests this (for example
> > enhancing the C++ library to consume structures produced this way).
>
> I think we are creating a C-based in-memory representation of Arrow
> (significantly simpler than what we have in C++, which involves smart
> pointers and other C++ concepts) and how people use these structs is
> up to them.
>
> > As I said, I really want to hear more opinions. Running this past various
> > developers I know, many have echoed my concerns but that really doesn't
> > matter (and who knows how much of that is colored by my presentation of the
> > issue). What do people here think? If someone builds an "Arrow" library
> > that implements this set of structures, how does one use it in Node? In
> > Java? Does it drive creation of a secondary set of interfaces in each of
> > those languages to work with this kind of pattern? (For example, in a JVM
> > view of the world, working with a plain struct in java rather than a set of
> > memory pointers against our existing IPC formats would be quite painful and
> > we'd definitely need to create some glue code for users. I worry the same
> > pattern would occur in many other languages.)
> >
>
> I'm fine to wait for more opinions, but I don't think that creating a
> strict C programming interface means that all languages have to figure
> out how to do FFI with it.
>
> > To respond directly to some of Wes's most recent comments from the email
> > below. I struggle to map your description of the situation to the rest of
> > the thread and the proposed patch.  For example, you say that a non-goal is
> > "creating a new canonical way to serialize metadata" bute the patch
> > proposes a concrete string based encoding system to describe data types.
> > Aren't those things in conflict?
> >
>
> Each language implementation represents in-memory schemas in a
> different way. In C++ we have the arrow::DataType classes. If the goal
> is to create a very compact C-based data model or Arrow, why is using
> a string representation of types instead of a more verbose object
> model inappropriate?
>

FWIW, the string-style representation of types is widespread. It's
used in the so-called C "buffer protocol" in Python and the "struct"
standard library module

https://docs.python.org/3/library/struct.html#module-struct

> > I'll also think more on this and challenge my own perspective. This isn't
> > where my focus is so my comments aren't as developed/thoughtful as I'd like.
> >
> >
> > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi Jacques,
> > >
> > > I think we've veered off course a bit and maybe we could reframe the
> > > discussion.
> > >
> > > Goals
> > > * A "drop-in" header-only C file that projects can use as a
> > > programming interface either internally only or to expose in-memory
> > > data structures between C functions at call sites. Ideally little to
> > > no disassembly/reassembly should be required on either "side" of the
> > > call site.
> > > * Simplifying adoption of Arrow for C programmers, or languages based
> > > around C FFI
> > >
> > > Non-goals
> > > * Expanding the columnar format or creating an alternative canonical
> > > in-memory representation
> > > * Creating a new canonical way to serialize metadata
> > >
> > > Note that this use case has been on my mind for more than 2 years:
> > > https://issues.apache.org/jira/browse/ARROW-1058
> > >
> > > I think there are a couple of potentially misleading things at play here
> > >
> > > 1. The use of the word "protocol". In C, a struct has a well-defined
> > > binary layout, so a C API is also an ABI. Using C structs to
> > > communicate data can be considered to be a protocol, but it means
> > > something different in the context of the "Arrow protocol". I think we
> > > need to call this a "C API"
> > >
> > > 2. The documentation for this in Antoine's PR is in the format/
> > > directory. It would probably be better to have a "C API" section in
> > > the documentation.
> > >
> > > The header file under discussion and the documentation about it is
> > > best considered as a "library".
> > >
> > > It might be useful at some point to create a C99 implementation of the
> > > IPC protocol as well using FlatCC with the goal of having a complete
> > > implementation of the columnar format in C with minimal binary
> > > footprint. This is analogous to the NanoPB project which is an
> > > implementation of Protocol Buffers with small code size
> > >
> > > https://github.com/nanopb/nanopb
> > >
> > > Let me know if this makes more sense.
> > >
> > > I think it's important to communicate clearly about this primarily for
> > > the benefit of the outside world which can confuse easily as we have
> > > observed over the last few years =)
> > >
> > > Wes
> > >
> > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org> wrote:
> > > >
> > > > I disagree with this statement:
> > > >
> > > > - the IPC format is meant for serialization while the C data protocol is
> > > > meants for in-memory communication, so different concerns apply
> > > >
> > > > If that is how the a particular implementation presents it, that is a
> > > > weaknesses of the implementation, not the format. The primary use case I
> > > > was focused on when working on the initial format was communication
> > > within
> > > > the same process. It seems like this is being used as a basis for the
> > > > introduction of new things when the premise is inconsistent with the
> > > > intention of the creation. The specific reason we used flatbuffers in the
> > > > project was to collapse the separation of in-process and out-of-process
> > > > communication. It means the same thing it does with the Arrow data
> > > itself:
> > > > that a consumer doesn't have to use a particular library to interact with
> > > > and use the data.
> > > >
> > > > It seems like there are two ideas here:
> > > >
> > > > 1) How do we make it easier for people to use Arrow?
> > > > 2) Should we implement a new in memory representation of Arrow that is
> > > > language specific.
> > > >
> > > > I'm entirely in support of number one. If for a particular type of
> > > domain,
> > > > people want an easier way to interact with Arrow, let's make a new
> > > library
> > > > that helps with that. In easy of our current libraries, we do many things
> > > > to make it easier to work with Arrow. None of those require a change to
> > > the
> > > > core format or are formalized as a new in-memory standard. The in-memory
> > > > representation of rust or javascript or java objects are implementation
> > > > details.
> > > >
> > > > I'm against number two as it creates a fragmentation problem. Arrow is
> > > > about having a single canonical format for memory for both metadata and
> > > > data. Having multiple in-memory formats (especially when some are not
> > > > language independent) is counter to the goals of the project.
> > >
> > > I don't think anyone is proposing anything that would cause fragmentation.
> > >
> > > A central question is whether it is useful to define a reusable C ABI
> > > for the Arrow columnar format, and if there is sufficient interest, a
> > > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> > > message) that assembles and disassembles the data structures defined
> > > in the C ABI.
> > >
> > > We could separately create a tiny implementation of the Arrow IPC
> > > protocol using FlatCC that could be dropped into applications
> > > requiring only a C compiler and nothing else.
> > >
> > >
> > > >
> > > > Two other, separate comments:
> > > > 1) I don't understand the idea that we need to change the way Arrow
> > > > fundamentally works so that people can avoid using a dependency. If the
> > > > dependency is small, open source and easy to build, people can fork it
> > > and
> > > > include directly if they want to. Let's not violate project principles
> > > > because DuckDB has a religious perspective on dependencies. If the
> > > problem
> > > > is people have to swallow too large of a pill to do basic things with
> > > Arrow
> > > > in C, let's focus on fixing that (to our definition of ease, not someone
> > > > else's). If FlatCC solves some those things, great. If we need to build a
> > > > baby integration library that is more C centric, great. Neither of those
> > > > things require implementing something at the format level.
> > > >
> > > > 2) It seems like we should discuss the data structure problem separately
> > > > from the reference management concern.
> > > >
> > > >
> > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > > hi Antoine,
> > > > >
> > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org>
> > > wrote:
> > > > > >
> > > > > >
> > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > > A couple things:
> > > > > > >
> > > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> > > better
> > > > > > > to have the same "shape" as an assembled array. Note that the C
> > > > > > > structs here have very nearly the same "shape" as the data
> > > structure
> > > > > > > representing a C++ Array object [1]. The disassembly and reassembly
> > > > > > > here is substantially simpler than the IPC protocol. A recursive
> > > > > > > structure in Flatbuffers would make RecordBatch messages much
> > > larger,
> > > > > > > so the flattened / disassembled representation we use for
> > > serialized
> > > > > > > record batches is the correct one
> > > > > >
> > > > > > I'm not sure I agree:
> > > > > >
> > > > > > - indeed, it's not a coincidence that the ArrowArray struct looks
> > > quite
> > > > > > closely like the C++ ArrayData object :-)  We have good experience
> > > with
> > > > > > that abstraction and it has proven to work quite well
> > > > > >
> > > > > > - the IPC format is meant for serialization while the C data
> > > protocol is
> > > > > > meants for in-memory communication, so different concerns apply
> > > > > >
> > > > > > - the fact that this makes the layout slightly larger doesn't seem
> > > > > > important at all; we're not talking about transferring data over the
> > > wire
> > > > > >
> > > > > > There's also another argument for having a recursive struct: it
> > > > > > simplifies how the data type is represented, since we can encode each
> > > > > > child type individually instead of encoding it in the parent's format
> > > > > > string (same applies for metadata and individual flags).
> > > > > >
> > > > >
> > > > > I was saying something different here. I was making an argument about
> > > > > why we use the flattened array-of-structs in the IPC protocol. One
> > > > > reason is that it's a more compact representation. That is not very
> > > > > important here because this protocol is only for *in-process* (for
> > > > > languages that have a C FFI facility) rather than *inter-process*
> > > > > communication.
> > > > >
> > > > > I agree also that the type encoding is simple, here, too, since we
> > > > > aren't having to split the schema and record batch between different
> > > > > serialized messages. There is some potential waste with having to
> > > > > populate the type fields multiple times when communicating a sequence
> > > > > of "chunks" from the same logical dataset.
> > > > >
> > > > > > > * The "formal" C protocol having the "assembled" shape means that
> > > many
> > > > > > > minimal Arrow users won't have to implement any separate data
> > > > > > > structures. They can just use the C struct directly or a slightly
> > > > > > > wrapped version thereof with some convenience functions.
> > > > > >
> > > > > > Yes, but the same applies to the current proposal.
> > > > > >
> > > > > > > * I think that requiring building a Flatbuffer for minimal use
> > > cases
> > > > > > > (e.g. communicating simple record batches with primitive types)
> > > passes
> > > > > > > on implementation burden to minimal users.
> > > > > >
> > > > > > It certainly does.
> > > > > >
> > > > > > > I think the mantra of the C protocol should be the following:
> > > > > > >
> > > > > > > * Users of the protocol have to write little to no code to use it.
> > > For
> > > > > > > example, populating an INT32 array should require only a few lines
> > > of
> > > > > > > code
> > > > > >
> > > > > > Agreed.  As a sidenote, the spec should have an example of doing
> > > this in
> > > > > > raw C.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Antoine.
> > > > >
> > >

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

On Wed, Oct 2, 2019 at 7:46 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> I'd like to hear more opinions from others on this topic. This conversation
> seems mostly dominated by comments from myself, Wes and Antoine.
>
> I think it is reasonable to argue that keeping any ABI (or header/struct
> pattern) as narrow as possible would allow us to minimize overlap with the
> existing in-memory specification. In Arrow's case, this could be as simple
> as a single memory pointer for schema (backed by flatbuffers) and a single
> memory location for data (that references the record batch header, which in
> turn provides pointers into the actual arrow data). Extensions would need
> to be added for reference management as done here but I continue to think
> we should defer discussion of that until the base data structures are
> resolved. I see the comments here as arguing for a much broader ABI, in
> part to support having people build "Arrow" components that interconnect
> using this new interface. I understand the desire to expand the ABI to be
> driven by needs to reduce dependencies and ease usability.
>
> The representation within the related patch is being presented as a way for
> applications to share Arrow data but is not easily accessible to all
> languages. I want to avoid a situation where someone says "I produced an
> Arrow API" when what they've really done is created a C interface which
> only a small subset of languages can actually leverage. For example, every
> language now knows how to parse the existing schema definition as rendered
> in flatbuf. In order to interact with something that implements this new
> pattern one would also be required to implement completely new schema
> consumption code. In the proposal itself it suggests this (for example
> enhancing the C++ library to consume structures produced this way).

I think we are creating a C-based in-memory representation of Arrow
(significantly simpler than what we have in C++, which involves smart
pointers and other C++ concepts) and how people use these structs is
up to them.

> As I said, I really want to hear more opinions. Running this past various
> developers I know, many have echoed my concerns but that really doesn't
> matter (and who knows how much of that is colored by my presentation of the
> issue). What do people here think? If someone builds an "Arrow" library
> that implements this set of structures, how does one use it in Node? In
> Java? Does it drive creation of a secondary set of interfaces in each of
> those languages to work with this kind of pattern? (For example, in a JVM
> view of the world, working with a plain struct in java rather than a set of
> memory pointers against our existing IPC formats would be quite painful and
> we'd definitely need to create some glue code for users. I worry the same
> pattern would occur in many other languages.)
>

I'm fine to wait for more opinions, but I don't think that creating a
strict C programming interface means that all languages have to figure
out how to do FFI with it.

> To respond directly to some of Wes's most recent comments from the email
> below. I struggle to map your description of the situation to the rest of
> the thread and the proposed patch.  For example, you say that a non-goal is
> "creating a new canonical way to serialize metadata" bute the patch
> proposes a concrete string based encoding system to describe data types.
> Aren't those things in conflict?
>

Each language implementation represents in-memory schemas in a
different way. In C++ we have the arrow::DataType classes. If the goal
is to create a very compact C-based data model or Arrow, why is using
a string representation of types instead of a more verbose object
model inappropriate?

> I'll also think more on this and challenge my own perspective. This isn't
> where my focus is so my comments aren't as developed/thoughtful as I'd like.
>
>
> On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Jacques,
> >
> > I think we've veered off course a bit and maybe we could reframe the
> > discussion.
> >
> > Goals
> > * A "drop-in" header-only C file that projects can use as a
> > programming interface either internally only or to expose in-memory
> > data structures between C functions at call sites. Ideally little to
> > no disassembly/reassembly should be required on either "side" of the
> > call site.
> > * Simplifying adoption of Arrow for C programmers, or languages based
> > around C FFI
> >
> > Non-goals
> > * Expanding the columnar format or creating an alternative canonical
> > in-memory representation
> > * Creating a new canonical way to serialize metadata
> >
> > Note that this use case has been on my mind for more than 2 years:
> > https://issues.apache.org/jira/browse/ARROW-1058
> >
> > I think there are a couple of potentially misleading things at play here
> >
> > 1. The use of the word "protocol". In C, a struct has a well-defined
> > binary layout, so a C API is also an ABI. Using C structs to
> > communicate data can be considered to be a protocol, but it means
> > something different in the context of the "Arrow protocol". I think we
> > need to call this a "C API"
> >
> > 2. The documentation for this in Antoine's PR is in the format/
> > directory. It would probably be better to have a "C API" section in
> > the documentation.
> >
> > The header file under discussion and the documentation about it is
> > best considered as a "library".
> >
> > It might be useful at some point to create a C99 implementation of the
> > IPC protocol as well using FlatCC with the goal of having a complete
> > implementation of the columnar format in C with minimal binary
> > footprint. This is analogous to the NanoPB project which is an
> > implementation of Protocol Buffers with small code size
> >
> > https://github.com/nanopb/nanopb
> >
> > Let me know if this makes more sense.
> >
> > I think it's important to communicate clearly about this primarily for
> > the benefit of the outside world which can confuse easily as we have
> > observed over the last few years =)
> >
> > Wes
> >
> > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org> wrote:
> > >
> > > I disagree with this statement:
> > >
> > > - the IPC format is meant for serialization while the C data protocol is
> > > meants for in-memory communication, so different concerns apply
> > >
> > > If that is how the a particular implementation presents it, that is a
> > > weaknesses of the implementation, not the format. The primary use case I
> > > was focused on when working on the initial format was communication
> > within
> > > the same process. It seems like this is being used as a basis for the
> > > introduction of new things when the premise is inconsistent with the
> > > intention of the creation. The specific reason we used flatbuffers in the
> > > project was to collapse the separation of in-process and out-of-process
> > > communication. It means the same thing it does with the Arrow data
> > itself:
> > > that a consumer doesn't have to use a particular library to interact with
> > > and use the data.
> > >
> > > It seems like there are two ideas here:
> > >
> > > 1) How do we make it easier for people to use Arrow?
> > > 2) Should we implement a new in memory representation of Arrow that is
> > > language specific.
> > >
> > > I'm entirely in support of number one. If for a particular type of
> > domain,
> > > people want an easier way to interact with Arrow, let's make a new
> > library
> > > that helps with that. In easy of our current libraries, we do many things
> > > to make it easier to work with Arrow. None of those require a change to
> > the
> > > core format or are formalized as a new in-memory standard. The in-memory
> > > representation of rust or javascript or java objects are implementation
> > > details.
> > >
> > > I'm against number two as it creates a fragmentation problem. Arrow is
> > > about having a single canonical format for memory for both metadata and
> > > data. Having multiple in-memory formats (especially when some are not
> > > language independent) is counter to the goals of the project.
> >
> > I don't think anyone is proposing anything that would cause fragmentation.
> >
> > A central question is whether it is useful to define a reusable C ABI
> > for the Arrow columnar format, and if there is sufficient interest, a
> > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> > message) that assembles and disassembles the data structures defined
> > in the C ABI.
> >
> > We could separately create a tiny implementation of the Arrow IPC
> > protocol using FlatCC that could be dropped into applications
> > requiring only a C compiler and nothing else.
> >
> >
> > >
> > > Two other, separate comments:
> > > 1) I don't understand the idea that we need to change the way Arrow
> > > fundamentally works so that people can avoid using a dependency. If the
> > > dependency is small, open source and easy to build, people can fork it
> > and
> > > include directly if they want to. Let's not violate project principles
> > > because DuckDB has a religious perspective on dependencies. If the
> > problem
> > > is people have to swallow too large of a pill to do basic things with
> > Arrow
> > > in C, let's focus on fixing that (to our definition of ease, not someone
> > > else's). If FlatCC solves some those things, great. If we need to build a
> > > baby integration library that is more C centric, great. Neither of those
> > > things require implementing something at the format level.
> > >
> > > 2) It seems like we should discuss the data structure problem separately
> > > from the reference management concern.
> > >
> > >
> > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > hi Antoine,
> > > >
> > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > > > >
> > > > >
> > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > A couple things:
> > > > > >
> > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> > better
> > > > > > to have the same "shape" as an assembled array. Note that the C
> > > > > > structs here have very nearly the same "shape" as the data
> > structure
> > > > > > representing a C++ Array object [1]. The disassembly and reassembly
> > > > > > here is substantially simpler than the IPC protocol. A recursive
> > > > > > structure in Flatbuffers would make RecordBatch messages much
> > larger,
> > > > > > so the flattened / disassembled representation we use for
> > serialized
> > > > > > record batches is the correct one
> > > > >
> > > > > I'm not sure I agree:
> > > > >
> > > > > - indeed, it's not a coincidence that the ArrowArray struct looks
> > quite
> > > > > closely like the C++ ArrayData object :-)  We have good experience
> > with
> > > > > that abstraction and it has proven to work quite well
> > > > >
> > > > > - the IPC format is meant for serialization while the C data
> > protocol is
> > > > > meants for in-memory communication, so different concerns apply
> > > > >
> > > > > - the fact that this makes the layout slightly larger doesn't seem
> > > > > important at all; we're not talking about transferring data over the
> > wire
> > > > >
> > > > > There's also another argument for having a recursive struct: it
> > > > > simplifies how the data type is represented, since we can encode each
> > > > > child type individually instead of encoding it in the parent's format
> > > > > string (same applies for metadata and individual flags).
> > > > >
> > > >
> > > > I was saying something different here. I was making an argument about
> > > > why we use the flattened array-of-structs in the IPC protocol. One
> > > > reason is that it's a more compact representation. That is not very
> > > > important here because this protocol is only for *in-process* (for
> > > > languages that have a C FFI facility) rather than *inter-process*
> > > > communication.
> > > >
> > > > I agree also that the type encoding is simple, here, too, since we
> > > > aren't having to split the schema and record batch between different
> > > > serialized messages. There is some potential waste with having to
> > > > populate the type fields multiple times when communicating a sequence
> > > > of "chunks" from the same logical dataset.
> > > >
> > > > > > * The "formal" C protocol having the "assembled" shape means that
> > many
> > > > > > minimal Arrow users won't have to implement any separate data
> > > > > > structures. They can just use the C struct directly or a slightly
> > > > > > wrapped version thereof with some convenience functions.
> > > > >
> > > > > Yes, but the same applies to the current proposal.
> > > > >
> > > > > > * I think that requiring building a Flatbuffer for minimal use
> > cases
> > > > > > (e.g. communicating simple record batches with primitive types)
> > passes
> > > > > > on implementation burden to minimal users.
> > > > >
> > > > > It certainly does.
> > > > >
> > > > > > I think the mantra of the C protocol should be the following:
> > > > > >
> > > > > > * Users of the protocol have to write little to no code to use it.
> > For
> > > > > > example, populating an INT32 array should require only a few lines
> > of
> > > > > > code
> > > > >
> > > > > Agreed.  As a sidenote, the spec should have an example of doing
> > this in
> > > > > raw C.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > >
> >

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

On Tue, Oct 8, 2019 at 3:34 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi Jacques,
>
> On Tue, Oct 8, 2019 at 1:54 PM Jacques Nadeau <ja...@apache.org> wrote:
> >
> > I removing all my objections to this work.
> >
> > I wish there was more feedback from additional community members. I continue to be concerned about fragmentation. I don't agree with the arguments here that we need to add a new api to make it easy for people to *not* use Arrow codebase. It seems like a punt on building useful libraries within the project that will ultimately hurt the interoperability story.
> >
>
> I think we'll have to take a "wait and see" approach. I believe the
> community needs to build accessible libraries that offer value to
> third party users, and we will continue to do that. I think there are
> use cases here that fall outside of which library to use, but time
> will tell.
>
> > As a side note, it seems like much of this is about people's distaste for flatbuffers. I know I regret using it. If we had a chance to do it over again, I would have chosen to use protobuf for everything except the data header, where I would hand write the encoding (since it is so simple anyway). If it is such a problem that people are contorting to work around it, maybe we should address that? Just a thought.
> >
>
> I think that using an Protobuf-like with IDL and a compiler presents a problem.

To clarify some inarticulate language since people reading may misinterpret.

Using an IDL-based metadata representation _in this C API_ presents a
potential roadblock for users.

As a canonical metadata representation with backward and forward
compatibility guarantees, it would be ill-advised to not use
Protobuf/Flatbuffers/Thrift

> Note that Flatbuffers is much better for C/C++ programmers and I still
> think it was the right choice for the project. Unlike Flatbuffers,
> C/C++ applications must either link libprotobuf.so or libprotobuf.a.
> Flatbuffers in C++ is a header-only dependency that is trivial to
> bundle with a project [1]. The same is true for Thrift, and this came
> up in the TF discussion [2]
>
> [1]: https://github.com/apache/arrow/tree/master/cpp/thirdparty/flatbuffers/include/flatbuffers
> [2]: https://github.com/tensorflow/community/pull/162#discussion_r332610486
>
> > Thanks for the discourse and patience.
> >
> > On Wed, Oct 2, 2019 at 10:12 PM Micah Kornfield <em...@gmail.com> wrote:
> >>
> >> Hi Wes,
> >> I agree for third-parties "A" (Field data structures) is the most useful.
> >>
> >> At least in my mind the discussion was for both first and third-parties.  I
> >> was trying to point out that "A" is less necessary as a first step for
> >> first-party integrations and could potentially require more effort if we
> >> already have the code that does "B" (field reassembly).
> >>
> >> Thanks,
> >> Micah
> >>
> >> On Wed, Oct 2, 2019 at 10:28 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >> > On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com>
> >> > wrote:
> >> > >
> >> > > I've tried to summarize my understanding of the debate so far and give
> >> > some
> >> > > initial thoughts. I think there are two potentially different sets of
> >> > users
> >> > > that we are targeting with a stable C API/ABI ourselves and external
> >> > > parties.
> >> > >
> >> > > 1.  Different language implementations within the Arrow project that want
> >> > > to call into each other's code.  We still don't have a great story around
> >> > > this in terms of reusable libraries and questions like [1] are a
> >> > motivating
> >> > > examples of making something better in this context.
> >> > > 2.  third-parties wishing to support/integrate with Arrow.  Some
> >> > > conjectures about these users:
> >> > >   - Users in this group are NOT necessarily familiar with existing
> >> > > technologies Arrow uses (i.e. flatbuffers)
> >> > >   - The stability of the API is the primary concern (consumers don't want
> >> > > to change when a new version of the library ships)
> >> > >   - An important secondary concern is additional libraries that need to
> >> > be
> >> > > integrated in addition to the API
> >> > >
> >> > > The main debate points seems to be:
> >> > >
> >> > > 1.  Vector/Array oriented API vs existing Record Batch.  Will an
> >> > additional
> >> > > column oriented API become too much of a maintenance headache/cause
> >> > > fragmentation?
> >> > >
> >> > >  - In my mind the question here is which set of users we are
> >> > prioritizing.
> >> > > IMO the combination of flatbuffers and translation to/from RecordBatch
> >> > > format offers too much friction to make it easy for a third-party
> >> > > implementer to use. If we are prioritizing for our own internal
> >> > use-cases I
> >> > > think we should try out a RecordBatch+Flatbuffers based C-API. We already
> >> > > have all the necessary building blocks.
> >> > >
> >> >
> >> > If a C function passes you a string containing a RecordBatch
> >> > Flatbuffers message, what happens next? This message has to be
> >> > reassembled into a recursive data structure before you can "do"
> >> > anything with it. Are we expecting every third party project to
> >> > implement:
> >> >
> >> > A. Data structures appropriate to represent a logical "field" in a
> >> > record batch (which have to be recursive to account for nested types'
> >> > children)
> >> > B. The logic to convert from the flattened Flatbuffers representation
> >> > to some implementation of A
> >> >
> >> > I'm arguing that we should provide both to third parties. To build B,
> >> > you need A. Some consumers will only use A. This discussion is
> >> > essentially about developing an ultraminimalist "drop-in" C
> >> > implementation of A.
> >> >
> >> > > 2.  How onerous is the dependency on flat-buffers both from a learning
> >> > > curve perspective and as dependency for third-party integrators?
> >> > > - Flatbuffers aren't entirely straight-forward and I think if we do move
> >> > > forward with an API based on Column/Array we should consider alternatives
> >> > > as long as the necessary parsing code can be done in a small amount of
> >> > code
> >> > > (I'm personally against JSON for this, but can see the arguments for it).
> >> > >
> >> > > 3.  Do all existing library implementations need to support both
> >> > > Column/Array a ABI?  How will compliance be checked for the new API/ABI?
> >> > >
> >> > > - I'm still thinking this through.
> >> > >
> >> > > [1]
> >> > >
> >> > https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
> >> > >
> >> > > On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org>
> >> > wrote:
> >> > >
> >> > > > I'd like to hear more opinions from others on this topic. This
> >> > conversation
> >> > > > seems mostly dominated by comments from myself, Wes and Antoine.
> >> > > >
> >> > > > I think it is reasonable to argue that keeping any ABI (or
> >> > header/struct
> >> > > > pattern) as narrow as possible would allow us to minimize overlap with
> >> > the
> >> > > > existing in-memory specification. In Arrow's case, this could be as
> >> > simple
> >> > > > as a single memory pointer for schema (backed by flatbuffers) and a
> >> > single
> >> > > > memory location for data (that references the record batch header,
> >> > which in
> >> > > > turn provides pointers into the actual arrow data). Extensions would
> >> > need
> >> > > > to be added for reference management as done here but I continue to
> >> > think
> >> > > > we should defer discussion of that until the base data structures are
> >> > > > resolved. I see the comments here as arguing for a much broader ABI, in
> >> > > > part to support having people build "Arrow" components that
> >> > interconnect
> >> > > > using this new interface. I understand the desire to expand the ABI to
> >> > be
> >> > > > driven by needs to reduce dependencies and ease usability.
> >> > > >
> >> > > > The representation within the related patch is being presented as a
> >> > way for
> >> > > > applications to share Arrow data but is not easily accessible to all
> >> > > > languages. I want to avoid a situation where someone says "I produced
> >> > an
> >> > > > Arrow API" when what they've really done is created a C interface which
> >> > > > only a small subset of languages can actually leverage. For example,
> >> > every
> >> > > > language now knows how to parse the existing schema definition as
> >> > rendered
> >> > > > in flatbuf. In order to interact with something that implements this
> >> > new
> >> > > > pattern one would also be required to implement completely new schema
> >> > > > consumption code. In the proposal itself it suggests this (for example
> >> > > > enhancing the C++ library to consume structures produced this way).
> >> > > >
> >> > > > As I said, I really want to hear more opinions. Running this past
> >> > various
> >> > > > developers I know, many have echoed my concerns but that really doesn't
> >> > > > matter (and who knows how much of that is colored by my presentation
> >> > of the
> >> > > > issue). What do people here think? If someone builds an "Arrow" library
> >> > > > that implements this set of structures, how does one use it in Node? In
> >> > > > Java? Does it drive creation of a secondary set of interfaces in each
> >> > of
> >> > > > those languages to work with this kind of pattern? (For example, in a
> >> > JVM
> >> > > > view of the world, working with a plain struct in java rather than a
> >> > set of
> >> > > > memory pointers against our existing IPC formats would be quite
> >> > painful and
> >> > > > we'd definitely need to create some glue code for users. I worry the
> >> > same
> >> > > > pattern would occur in many other languages.)
> >> > > >
> >> > > > To respond directly to some of Wes's most recent comments from the
> >> > email
> >> > > > below. I struggle to map your description of the situation to the rest
> >> > of
> >> > > > the thread and the proposed patch.  For example, you say that a
> >> > non-goal is
> >> > > > "creating a new canonical way to serialize metadata" bute the patch
> >> > > > proposes a concrete string based encoding system to describe data
> >> > types.
> >> > > > Aren't those things in conflict?
> >> > > >
> >> > > > I'll also think more on this and challenge my own perspective. This
> >> > isn't
> >> > > > where my focus is so my comments aren't as developed/thoughtful as I'd
> >> > > > like.
> >> > > >
> >> > > >
> >> > > > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > > hi Jacques,
> >> > > > >
> >> > > > > I think we've veered off course a bit and maybe we could reframe the
> >> > > > > discussion.
> >> > > > >
> >> > > > > Goals
> >> > > > > * A "drop-in" header-only C file that projects can use as a
> >> > > > > programming interface either internally only or to expose in-memory
> >> > > > > data structures between C functions at call sites. Ideally little to
> >> > > > > no disassembly/reassembly should be required on either "side" of the
> >> > > > > call site.
> >> > > > > * Simplifying adoption of Arrow for C programmers, or languages based
> >> > > > > around C FFI
> >> > > > >
> >> > > > > Non-goals
> >> > > > > * Expanding the columnar format or creating an alternative canonical
> >> > > > > in-memory representation
> >> > > > > * Creating a new canonical way to serialize metadata
> >> > > > >
> >> > > > > Note that this use case has been on my mind for more than 2 years:
> >> > > > > https://issues.apache.org/jira/browse/ARROW-1058
> >> > > > >
> >> > > > > I think there are a couple of potentially misleading things at play
> >> > here
> >> > > > >
> >> > > > > 1. The use of the word "protocol". In C, a struct has a well-defined
> >> > > > > binary layout, so a C API is also an ABI. Using C structs to
> >> > > > > communicate data can be considered to be a protocol, but it means
> >> > > > > something different in the context of the "Arrow protocol". I think
> >> > we
> >> > > > > need to call this a "C API"
> >> > > > >
> >> > > > > 2. The documentation for this in Antoine's PR is in the format/
> >> > > > > directory. It would probably be better to have a "C API" section in
> >> > > > > the documentation.
> >> > > > >
> >> > > > > The header file under discussion and the documentation about it is
> >> > > > > best considered as a "library".
> >> > > > >
> >> > > > > It might be useful at some point to create a C99 implementation of
> >> > the
> >> > > > > IPC protocol as well using FlatCC with the goal of having a complete
> >> > > > > implementation of the columnar format in C with minimal binary
> >> > > > > footprint. This is analogous to the NanoPB project which is an
> >> > > > > implementation of Protocol Buffers with small code size
> >> > > > >
> >> > > > > https://github.com/nanopb/nanopb
> >> > > > >
> >> > > > > Let me know if this makes more sense.
> >> > > > >
> >> > > > > I think it's important to communicate clearly about this primarily
> >> > for
> >> > > > > the benefit of the outside world which can confuse easily as we have
> >> > > > > observed over the last few years =)
> >> > > > >
> >> > > > > Wes
> >> > > > >
> >> > > > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > I disagree with this statement:
> >> > > > > >
> >> > > > > > - the IPC format is meant for serialization while the C data
> >> > protocol
> >> > > > is
> >> > > > > > meants for in-memory communication, so different concerns apply
> >> > > > > >
> >> > > > > > If that is how the a particular implementation presents it, that
> >> > is a
> >> > > > > > weaknesses of the implementation, not the format. The primary use
> >> > case
> >> > > > I
> >> > > > > > was focused on when working on the initial format was communication
> >> > > > > within
> >> > > > > > the same process. It seems like this is being used as a basis for
> >> > the
> >> > > > > > introduction of new things when the premise is inconsistent with
> >> > the
> >> > > > > > intention of the creation. The specific reason we used flatbuffers
> >> > in
> >> > > > the
> >> > > > > > project was to collapse the separation of in-process and
> >> > out-of-process
> >> > > > > > communication. It means the same thing it does with the Arrow data
> >> > > > > itself:
> >> > > > > > that a consumer doesn't have to use a particular library to
> >> > interact
> >> > > > with
> >> > > > > > and use the data.
> >> > > > > >
> >> > > > > > It seems like there are two ideas here:
> >> > > > > >
> >> > > > > > 1) How do we make it easier for people to use Arrow?
> >> > > > > > 2) Should we implement a new in memory representation of Arrow
> >> > that is
> >> > > > > > language specific.
> >> > > > > >
> >> > > > > > I'm entirely in support of number one. If for a particular type of
> >> > > > > domain,
> >> > > > > > people want an easier way to interact with Arrow, let's make a new
> >> > > > > library
> >> > > > > > that helps with that. In easy of our current libraries, we do many
> >> > > > things
> >> > > > > > to make it easier to work with Arrow. None of those require a
> >> > change to
> >> > > > > the
> >> > > > > > core format or are formalized as a new in-memory standard. The
> >> > > > in-memory
> >> > > > > > representation of rust or javascript or java objects are
> >> > implementation
> >> > > > > > details.
> >> > > > > >
> >> > > > > > I'm against number two as it creates a fragmentation problem.
> >> > Arrow is
> >> > > > > > about having a single canonical format for memory for both
> >> > metadata and
> >> > > > > > data. Having multiple in-memory formats (especially when some are
> >> > not
> >> > > > > > language independent) is counter to the goals of the project.
> >> > > > >
> >> > > > > I don't think anyone is proposing anything that would cause
> >> > > > fragmentation.
> >> > > > >
> >> > > > > A central question is whether it is useful to define a reusable C ABI
> >> > > > > for the Arrow columnar format, and if there is sufficient interest, a
> >> > > > > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> >> > > > > message) that assembles and disassembles the data structures defined
> >> > > > > in the C ABI.
> >> > > > >
> >> > > > > We could separately create a tiny implementation of the Arrow IPC
> >> > > > > protocol using FlatCC that could be dropped into applications
> >> > > > > requiring only a C compiler and nothing else.
> >> > > > >
> >> > > > >
> >> > > > > >
> >> > > > > > Two other, separate comments:
> >> > > > > > 1) I don't understand the idea that we need to change the way Arrow
> >> > > > > > fundamentally works so that people can avoid using a dependency.
> >> > If the
> >> > > > > > dependency is small, open source and easy to build, people can
> >> > fork it
> >> > > > > and
> >> > > > > > include directly if they want to. Let's not violate project
> >> > principles
> >> > > > > > because DuckDB has a religious perspective on dependencies. If the
> >> > > > > problem
> >> > > > > > is people have to swallow too large of a pill to do basic things
> >> > with
> >> > > > > Arrow
> >> > > > > > in C, let's focus on fixing that (to our definition of ease, not
> >> > > > someone
> >> > > > > > else's). If FlatCC solves some those things, great. If we need to
> >> > > > build a
> >> > > > > > baby integration library that is more C centric, great. Neither of
> >> > > > those
> >> > > > > > things require implementing something at the format level.
> >> > > > > >
> >> > > > > > 2) It seems like we should discuss the data structure problem
> >> > > > separately
> >> > > > > > from the reference management concern.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > hi Antoine,
> >> > > > > > >
> >> > > > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <
> >> > antoine@python.org>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> >> > > > > > > > > A couple things:
> >> > > > > > > > >
> >> > > > > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> >> > > > > better
> >> > > > > > > > > to have the same "shape" as an assembled array. Note that
> >> > the C
> >> > > > > > > > > structs here have very nearly the same "shape" as the data
> >> > > > > structure
> >> > > > > > > > > representing a C++ Array object [1]. The disassembly and
> >> > > > reassembly
> >> > > > > > > > > here is substantially simpler than the IPC protocol. A
> >> > recursive
> >> > > > > > > > > structure in Flatbuffers would make RecordBatch messages much
> >> > > > > larger,
> >> > > > > > > > > so the flattened / disassembled representation we use for
> >> > > > > serialized
> >> > > > > > > > > record batches is the correct one
> >> > > > > > > >
> >> > > > > > > > I'm not sure I agree:
> >> > > > > > > >
> >> > > > > > > > - indeed, it's not a coincidence that the ArrowArray struct
> >> > looks
> >> > > > > quite
> >> > > > > > > > closely like the C++ ArrayData object :-)  We have good
> >> > experience
> >> > > > > with
> >> > > > > > > > that abstraction and it has proven to work quite well
> >> > > > > > > >
> >> > > > > > > > - the IPC format is meant for serialization while the C data
> >> > > > > protocol is
> >> > > > > > > > meants for in-memory communication, so different concerns apply
> >> > > > > > > >
> >> > > > > > > > - the fact that this makes the layout slightly larger doesn't
> >> > seem
> >> > > > > > > > important at all; we're not talking about transferring data
> >> > over
> >> > > > the
> >> > > > > wire
> >> > > > > > > >
> >> > > > > > > > There's also another argument for having a recursive struct: it
> >> > > > > > > > simplifies how the data type is represented, since we can
> >> > encode
> >> > > > each
> >> > > > > > > > child type individually instead of encoding it in the parent's
> >> > > > format
> >> > > > > > > > string (same applies for metadata and individual flags).
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > I was saying something different here. I was making an argument
> >> > about
> >> > > > > > > why we use the flattened array-of-structs in the IPC protocol.
> >> > One
> >> > > > > > > reason is that it's a more compact representation. That is not
> >> > very
> >> > > > > > > important here because this protocol is only for *in-process*
> >> > (for
> >> > > > > > > languages that have a C FFI facility) rather than *inter-process*
> >> > > > > > > communication.
> >> > > > > > >
> >> > > > > > > I agree also that the type encoding is simple, here, too, since
> >> > we
> >> > > > > > > aren't having to split the schema and record batch between
> >> > different
> >> > > > > > > serialized messages. There is some potential waste with having to
> >> > > > > > > populate the type fields multiple times when communicating a
> >> > sequence
> >> > > > > > > of "chunks" from the same logical dataset.
> >> > > > > > >
> >> > > > > > > > > * The "formal" C protocol having the "assembled" shape means
> >> > that
> >> > > > > many
> >> > > > > > > > > minimal Arrow users won't have to implement any separate data
> >> > > > > > > > > structures. They can just use the C struct directly or a
> >> > slightly
> >> > > > > > > > > wrapped version thereof with some convenience functions.
> >> > > > > > > >
> >> > > > > > > > Yes, but the same applies to the current proposal.
> >> > > > > > > >
> >> > > > > > > > > * I think that requiring building a Flatbuffer for minimal
> >> > use
> >> > > > > cases
> >> > > > > > > > > (e.g. communicating simple record batches with primitive
> >> > types)
> >> > > > > passes
> >> > > > > > > > > on implementation burden to minimal users.
> >> > > > > > > >
> >> > > > > > > > It certainly does.
> >> > > > > > > >
> >> > > > > > > > > I think the mantra of the C protocol should be the following:
> >> > > > > > > > >
> >> > > > > > > > > * Users of the protocol have to write little to no code to
> >> > use
> >> > > > it.
> >> > > > > For
> >> > > > > > > > > example, populating an INT32 array should require only a few
> >> > > > lines
> >> > > > > of
> >> > > > > > > > > code
> >> > > > > > > >
> >> > > > > > > > Agreed.  As a sidenote, the spec should have an example of
> >> > doing
> >> > > > > this in
> >> > > > > > > > raw C.
> >> > > > > > > >
> >> > > > > > > > Regards
> >> > > > > > > >
> >> > > > > > > > Antoine.
> >> > > > > > >
> >> > > > >
> >> > > >
> >> >

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

hi Jacques,

On Tue, Oct 8, 2019 at 1:54 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> I removing all my objections to this work.
>
> I wish there was more feedback from additional community members. I continue to be concerned about fragmentation. I don't agree with the arguments here that we need to add a new api to make it easy for people to *not* use Arrow codebase. It seems like a punt on building useful libraries within the project that will ultimately hurt the interoperability story.
>

I think we'll have to take a "wait and see" approach. I believe the
community needs to build accessible libraries that offer value to
third party users, and we will continue to do that. I think there are
use cases here that fall outside of which library to use, but time
will tell.

> As a side note, it seems like much of this is about people's distaste for flatbuffers. I know I regret using it. If we had a chance to do it over again, I would have chosen to use protobuf for everything except the data header, where I would hand write the encoding (since it is so simple anyway). If it is such a problem that people are contorting to work around it, maybe we should address that? Just a thought.
>

I think that using an Protobuf-like with IDL and a compiler presents a problem.

Note that Flatbuffers is much better for C/C++ programmers and I still
think it was the right choice for the project. Unlike Flatbuffers,
C/C++ applications must either link libprotobuf.so or libprotobuf.a.
Flatbuffers in C++ is a header-only dependency that is trivial to
bundle with a project [1]. The same is true for Thrift, and this came
up in the TF discussion [2]

[1]: https://github.com/apache/arrow/tree/master/cpp/thirdparty/flatbuffers/include/flatbuffers
[2]: https://github.com/tensorflow/community/pull/162#discussion_r332610486

> Thanks for the discourse and patience.
>
> On Wed, Oct 2, 2019 at 10:12 PM Micah Kornfield <em...@gmail.com> wrote:
>>
>> Hi Wes,
>> I agree for third-parties "A" (Field data structures) is the most useful.
>>
>> At least in my mind the discussion was for both first and third-parties.  I
>> was trying to point out that "A" is less necessary as a first step for
>> first-party integrations and could potentially require more effort if we
>> already have the code that does "B" (field reassembly).
>>
>> Thanks,
>> Micah
>>
>> On Wed, Oct 2, 2019 at 10:28 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> > On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com>
>> > wrote:
>> > >
>> > > I've tried to summarize my understanding of the debate so far and give
>> > some
>> > > initial thoughts. I think there are two potentially different sets of
>> > users
>> > > that we are targeting with a stable C API/ABI ourselves and external
>> > > parties.
>> > >
>> > > 1.  Different language implementations within the Arrow project that want
>> > > to call into each other's code.  We still don't have a great story around
>> > > this in terms of reusable libraries and questions like [1] are a
>> > motivating
>> > > examples of making something better in this context.
>> > > 2.  third-parties wishing to support/integrate with Arrow.  Some
>> > > conjectures about these users:
>> > >   - Users in this group are NOT necessarily familiar with existing
>> > > technologies Arrow uses (i.e. flatbuffers)
>> > >   - The stability of the API is the primary concern (consumers don't want
>> > > to change when a new version of the library ships)
>> > >   - An important secondary concern is additional libraries that need to
>> > be
>> > > integrated in addition to the API
>> > >
>> > > The main debate points seems to be:
>> > >
>> > > 1.  Vector/Array oriented API vs existing Record Batch.  Will an
>> > additional
>> > > column oriented API become too much of a maintenance headache/cause
>> > > fragmentation?
>> > >
>> > >  - In my mind the question here is which set of users we are
>> > prioritizing.
>> > > IMO the combination of flatbuffers and translation to/from RecordBatch
>> > > format offers too much friction to make it easy for a third-party
>> > > implementer to use. If we are prioritizing for our own internal
>> > use-cases I
>> > > think we should try out a RecordBatch+Flatbuffers based C-API. We already
>> > > have all the necessary building blocks.
>> > >
>> >
>> > If a C function passes you a string containing a RecordBatch
>> > Flatbuffers message, what happens next? This message has to be
>> > reassembled into a recursive data structure before you can "do"
>> > anything with it. Are we expecting every third party project to
>> > implement:
>> >
>> > A. Data structures appropriate to represent a logical "field" in a
>> > record batch (which have to be recursive to account for nested types'
>> > children)
>> > B. The logic to convert from the flattened Flatbuffers representation
>> > to some implementation of A
>> >
>> > I'm arguing that we should provide both to third parties. To build B,
>> > you need A. Some consumers will only use A. This discussion is
>> > essentially about developing an ultraminimalist "drop-in" C
>> > implementation of A.
>> >
>> > > 2.  How onerous is the dependency on flat-buffers both from a learning
>> > > curve perspective and as dependency for third-party integrators?
>> > > - Flatbuffers aren't entirely straight-forward and I think if we do move
>> > > forward with an API based on Column/Array we should consider alternatives
>> > > as long as the necessary parsing code can be done in a small amount of
>> > code
>> > > (I'm personally against JSON for this, but can see the arguments for it).
>> > >
>> > > 3.  Do all existing library implementations need to support both
>> > > Column/Array a ABI?  How will compliance be checked for the new API/ABI?
>> > >
>> > > - I'm still thinking this through.
>> > >
>> > > [1]
>> > >
>> > https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
>> > >
>> > > On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org>
>> > wrote:
>> > >
>> > > > I'd like to hear more opinions from others on this topic. This
>> > conversation
>> > > > seems mostly dominated by comments from myself, Wes and Antoine.
>> > > >
>> > > > I think it is reasonable to argue that keeping any ABI (or
>> > header/struct
>> > > > pattern) as narrow as possible would allow us to minimize overlap with
>> > the
>> > > > existing in-memory specification. In Arrow's case, this could be as
>> > simple
>> > > > as a single memory pointer for schema (backed by flatbuffers) and a
>> > single
>> > > > memory location for data (that references the record batch header,
>> > which in
>> > > > turn provides pointers into the actual arrow data). Extensions would
>> > need
>> > > > to be added for reference management as done here but I continue to
>> > think
>> > > > we should defer discussion of that until the base data structures are
>> > > > resolved. I see the comments here as arguing for a much broader ABI, in
>> > > > part to support having people build "Arrow" components that
>> > interconnect
>> > > > using this new interface. I understand the desire to expand the ABI to
>> > be
>> > > > driven by needs to reduce dependencies and ease usability.
>> > > >
>> > > > The representation within the related patch is being presented as a
>> > way for
>> > > > applications to share Arrow data but is not easily accessible to all
>> > > > languages. I want to avoid a situation where someone says "I produced
>> > an
>> > > > Arrow API" when what they've really done is created a C interface which
>> > > > only a small subset of languages can actually leverage. For example,
>> > every
>> > > > language now knows how to parse the existing schema definition as
>> > rendered
>> > > > in flatbuf. In order to interact with something that implements this
>> > new
>> > > > pattern one would also be required to implement completely new schema
>> > > > consumption code. In the proposal itself it suggests this (for example
>> > > > enhancing the C++ library to consume structures produced this way).
>> > > >
>> > > > As I said, I really want to hear more opinions. Running this past
>> > various
>> > > > developers I know, many have echoed my concerns but that really doesn't
>> > > > matter (and who knows how much of that is colored by my presentation
>> > of the
>> > > > issue). What do people here think? If someone builds an "Arrow" library
>> > > > that implements this set of structures, how does one use it in Node? In
>> > > > Java? Does it drive creation of a secondary set of interfaces in each
>> > of
>> > > > those languages to work with this kind of pattern? (For example, in a
>> > JVM
>> > > > view of the world, working with a plain struct in java rather than a
>> > set of
>> > > > memory pointers against our existing IPC formats would be quite
>> > painful and
>> > > > we'd definitely need to create some glue code for users. I worry the
>> > same
>> > > > pattern would occur in many other languages.)
>> > > >
>> > > > To respond directly to some of Wes's most recent comments from the
>> > email
>> > > > below. I struggle to map your description of the situation to the rest
>> > of
>> > > > the thread and the proposed patch.  For example, you say that a
>> > non-goal is
>> > > > "creating a new canonical way to serialize metadata" bute the patch
>> > > > proposes a concrete string based encoding system to describe data
>> > types.
>> > > > Aren't those things in conflict?
>> > > >
>> > > > I'll also think more on this and challenge my own perspective. This
>> > isn't
>> > > > where my focus is so my comments aren't as developed/thoughtful as I'd
>> > > > like.
>> > > >
>> > > >
>> > > > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com>
>> > wrote:
>> > > >
>> > > > > hi Jacques,
>> > > > >
>> > > > > I think we've veered off course a bit and maybe we could reframe the
>> > > > > discussion.
>> > > > >
>> > > > > Goals
>> > > > > * A "drop-in" header-only C file that projects can use as a
>> > > > > programming interface either internally only or to expose in-memory
>> > > > > data structures between C functions at call sites. Ideally little to
>> > > > > no disassembly/reassembly should be required on either "side" of the
>> > > > > call site.
>> > > > > * Simplifying adoption of Arrow for C programmers, or languages based
>> > > > > around C FFI
>> > > > >
>> > > > > Non-goals
>> > > > > * Expanding the columnar format or creating an alternative canonical
>> > > > > in-memory representation
>> > > > > * Creating a new canonical way to serialize metadata
>> > > > >
>> > > > > Note that this use case has been on my mind for more than 2 years:
>> > > > > https://issues.apache.org/jira/browse/ARROW-1058
>> > > > >
>> > > > > I think there are a couple of potentially misleading things at play
>> > here
>> > > > >
>> > > > > 1. The use of the word "protocol". In C, a struct has a well-defined
>> > > > > binary layout, so a C API is also an ABI. Using C structs to
>> > > > > communicate data can be considered to be a protocol, but it means
>> > > > > something different in the context of the "Arrow protocol". I think
>> > we
>> > > > > need to call this a "C API"
>> > > > >
>> > > > > 2. The documentation for this in Antoine's PR is in the format/
>> > > > > directory. It would probably be better to have a "C API" section in
>> > > > > the documentation.
>> > > > >
>> > > > > The header file under discussion and the documentation about it is
>> > > > > best considered as a "library".
>> > > > >
>> > > > > It might be useful at some point to create a C99 implementation of
>> > the
>> > > > > IPC protocol as well using FlatCC with the goal of having a complete
>> > > > > implementation of the columnar format in C with minimal binary
>> > > > > footprint. This is analogous to the NanoPB project which is an
>> > > > > implementation of Protocol Buffers with small code size
>> > > > >
>> > > > > https://github.com/nanopb/nanopb
>> > > > >
>> > > > > Let me know if this makes more sense.
>> > > > >
>> > > > > I think it's important to communicate clearly about this primarily
>> > for
>> > > > > the benefit of the outside world which can confuse easily as we have
>> > > > > observed over the last few years =)
>> > > > >
>> > > > > Wes
>> > > > >
>> > > > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
>> > > > wrote:
>> > > > > >
>> > > > > > I disagree with this statement:
>> > > > > >
>> > > > > > - the IPC format is meant for serialization while the C data
>> > protocol
>> > > > is
>> > > > > > meants for in-memory communication, so different concerns apply
>> > > > > >
>> > > > > > If that is how the a particular implementation presents it, that
>> > is a
>> > > > > > weaknesses of the implementation, not the format. The primary use
>> > case
>> > > > I
>> > > > > > was focused on when working on the initial format was communication
>> > > > > within
>> > > > > > the same process. It seems like this is being used as a basis for
>> > the
>> > > > > > introduction of new things when the premise is inconsistent with
>> > the
>> > > > > > intention of the creation. The specific reason we used flatbuffers
>> > in
>> > > > the
>> > > > > > project was to collapse the separation of in-process and
>> > out-of-process
>> > > > > > communication. It means the same thing it does with the Arrow data
>> > > > > itself:
>> > > > > > that a consumer doesn't have to use a particular library to
>> > interact
>> > > > with
>> > > > > > and use the data.
>> > > > > >
>> > > > > > It seems like there are two ideas here:
>> > > > > >
>> > > > > > 1) How do we make it easier for people to use Arrow?
>> > > > > > 2) Should we implement a new in memory representation of Arrow
>> > that is
>> > > > > > language specific.
>> > > > > >
>> > > > > > I'm entirely in support of number one. If for a particular type of
>> > > > > domain,
>> > > > > > people want an easier way to interact with Arrow, let's make a new
>> > > > > library
>> > > > > > that helps with that. In easy of our current libraries, we do many
>> > > > things
>> > > > > > to make it easier to work with Arrow. None of those require a
>> > change to
>> > > > > the
>> > > > > > core format or are formalized as a new in-memory standard. The
>> > > > in-memory
>> > > > > > representation of rust or javascript or java objects are
>> > implementation
>> > > > > > details.
>> > > > > >
>> > > > > > I'm against number two as it creates a fragmentation problem.
>> > Arrow is
>> > > > > > about having a single canonical format for memory for both
>> > metadata and
>> > > > > > data. Having multiple in-memory formats (especially when some are
>> > not
>> > > > > > language independent) is counter to the goals of the project.
>> > > > >
>> > > > > I don't think anyone is proposing anything that would cause
>> > > > fragmentation.
>> > > > >
>> > > > > A central question is whether it is useful to define a reusable C ABI
>> > > > > for the Arrow columnar format, and if there is sufficient interest, a
>> > > > > tiny C implementation of the IPC protocol (which uses the Flatbuffers
>> > > > > message) that assembles and disassembles the data structures defined
>> > > > > in the C ABI.
>> > > > >
>> > > > > We could separately create a tiny implementation of the Arrow IPC
>> > > > > protocol using FlatCC that could be dropped into applications
>> > > > > requiring only a C compiler and nothing else.
>> > > > >
>> > > > >
>> > > > > >
>> > > > > > Two other, separate comments:
>> > > > > > 1) I don't understand the idea that we need to change the way Arrow
>> > > > > > fundamentally works so that people can avoid using a dependency.
>> > If the
>> > > > > > dependency is small, open source and easy to build, people can
>> > fork it
>> > > > > and
>> > > > > > include directly if they want to. Let's not violate project
>> > principles
>> > > > > > because DuckDB has a religious perspective on dependencies. If the
>> > > > > problem
>> > > > > > is people have to swallow too large of a pill to do basic things
>> > with
>> > > > > Arrow
>> > > > > > in C, let's focus on fixing that (to our definition of ease, not
>> > > > someone
>> > > > > > else's). If FlatCC solves some those things, great. If we need to
>> > > > build a
>> > > > > > baby integration library that is more C centric, great. Neither of
>> > > > those
>> > > > > > things require implementing something at the format level.
>> > > > > >
>> > > > > > 2) It seems like we should discuss the data structure problem
>> > > > separately
>> > > > > > from the reference management concern.
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > hi Antoine,
>> > > > > > >
>> > > > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <
>> > antoine@python.org>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
>> > > > > > > > > A couple things:
>> > > > > > > > >
>> > > > > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
>> > > > > better
>> > > > > > > > > to have the same "shape" as an assembled array. Note that
>> > the C
>> > > > > > > > > structs here have very nearly the same "shape" as the data
>> > > > > structure
>> > > > > > > > > representing a C++ Array object [1]. The disassembly and
>> > > > reassembly
>> > > > > > > > > here is substantially simpler than the IPC protocol. A
>> > recursive
>> > > > > > > > > structure in Flatbuffers would make RecordBatch messages much
>> > > > > larger,
>> > > > > > > > > so the flattened / disassembled representation we use for
>> > > > > serialized
>> > > > > > > > > record batches is the correct one
>> > > > > > > >
>> > > > > > > > I'm not sure I agree:
>> > > > > > > >
>> > > > > > > > - indeed, it's not a coincidence that the ArrowArray struct
>> > looks
>> > > > > quite
>> > > > > > > > closely like the C++ ArrayData object :-)  We have good
>> > experience
>> > > > > with
>> > > > > > > > that abstraction and it has proven to work quite well
>> > > > > > > >
>> > > > > > > > - the IPC format is meant for serialization while the C data
>> > > > > protocol is
>> > > > > > > > meants for in-memory communication, so different concerns apply
>> > > > > > > >
>> > > > > > > > - the fact that this makes the layout slightly larger doesn't
>> > seem
>> > > > > > > > important at all; we're not talking about transferring data
>> > over
>> > > > the
>> > > > > wire
>> > > > > > > >
>> > > > > > > > There's also another argument for having a recursive struct: it
>> > > > > > > > simplifies how the data type is represented, since we can
>> > encode
>> > > > each
>> > > > > > > > child type individually instead of encoding it in the parent's
>> > > > format
>> > > > > > > > string (same applies for metadata and individual flags).
>> > > > > > > >
>> > > > > > >
>> > > > > > > I was saying something different here. I was making an argument
>> > about
>> > > > > > > why we use the flattened array-of-structs in the IPC protocol.
>> > One
>> > > > > > > reason is that it's a more compact representation. That is not
>> > very
>> > > > > > > important here because this protocol is only for *in-process*
>> > (for
>> > > > > > > languages that have a C FFI facility) rather than *inter-process*
>> > > > > > > communication.
>> > > > > > >
>> > > > > > > I agree also that the type encoding is simple, here, too, since
>> > we
>> > > > > > > aren't having to split the schema and record batch between
>> > different
>> > > > > > > serialized messages. There is some potential waste with having to
>> > > > > > > populate the type fields multiple times when communicating a
>> > sequence
>> > > > > > > of "chunks" from the same logical dataset.
>> > > > > > >
>> > > > > > > > > * The "formal" C protocol having the "assembled" shape means
>> > that
>> > > > > many
>> > > > > > > > > minimal Arrow users won't have to implement any separate data
>> > > > > > > > > structures. They can just use the C struct directly or a
>> > slightly
>> > > > > > > > > wrapped version thereof with some convenience functions.
>> > > > > > > >
>> > > > > > > > Yes, but the same applies to the current proposal.
>> > > > > > > >
>> > > > > > > > > * I think that requiring building a Flatbuffer for minimal
>> > use
>> > > > > cases
>> > > > > > > > > (e.g. communicating simple record batches with primitive
>> > types)
>> > > > > passes
>> > > > > > > > > on implementation burden to minimal users.
>> > > > > > > >
>> > > > > > > > It certainly does.
>> > > > > > > >
>> > > > > > > > > I think the mantra of the C protocol should be the following:
>> > > > > > > > >
>> > > > > > > > > * Users of the protocol have to write little to no code to
>> > use
>> > > > it.
>> > > > > For
>> > > > > > > > > example, populating an INT32 array should require only a few
>> > > > lines
>> > > > > of
>> > > > > > > > > code
>> > > > > > > >
>> > > > > > > > Agreed.  As a sidenote, the spec should have an example of
>> > doing
>> > > > > this in
>> > > > > > > > raw C.
>> > > > > > > >
>> > > > > > > > Regards
>> > > > > > > >
>> > > > > > > > Antoine.
>> > > > > > >
>> > > > >
>> > > >
>> >

Re: [DISCUSS] C-level in-process array protocol

["Uwe L. Korn" <uw...@xhochy.com>]

I'm not sure whether flatbuffers is actually an issue in the end but keeping it out of the C-API definitely simplifies it a bit adoption-wise. I don't think that though that using protobuf would make a difference here.

In general, I really like the C-interface work as sadly C-APIs are still the most accessible ones. Even when using the official Arrow C++ library, I often want access to the underlying data with some other non-C++ processing library having the C-interface is making my life easier. In my case I'm working with Numba (LLVM-based JIT for a subset of numerical Python) and this is not easily supporting interfaces to C++ but can work with C FFI calls directly. 

Uwe

On Tue, Oct 8, 2019, at 8:54 PM, Jacques Nadeau wrote:
> I removing all my objections to this work.
> 
> I wish there was more feedback from additional community members. I
> continue to be concerned about fragmentation. I don't agree with the
> arguments here that we need to add a new api to make it easy for people to
> *not* use Arrow codebase. It seems like a punt on building useful libraries
> within the project that will ultimately hurt the interoperability story.
> 
> As a side note, it seems like much of this is about people's distaste for
> flatbuffers. I know I regret using it. If we had a chance to do it over
> again, I would have chosen to use protobuf for everything except the data
> header, where I would hand write the encoding (since it is so simple
> anyway). If it is such a problem that people are contorting to work around
> it, maybe we should address that? Just a thought.
> 
> Thanks for the discourse and patience.
> 
> On Wed, Oct 2, 2019 at 10:12 PM Micah Kornfield <em...@gmail.com>
> wrote:
> 
> > Hi Wes,
> > I agree for third-parties "A" (Field data structures) is the most useful.
> >
> > At least in my mind the discussion was for both first and third-parties.  I
> > was trying to point out that "A" is less necessary as a first step for
> > first-party integrations and could potentially require more effort if we
> > already have the code that does "B" (field reassembly).
> >
> > Thanks,
> > Micah
> >
> > On Wed, Oct 2, 2019 at 10:28 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > > >
> > > > I've tried to summarize my understanding of the debate so far and give
> > > some
> > > > initial thoughts. I think there are two potentially different sets of
> > > users
> > > > that we are targeting with a stable C API/ABI ourselves and external
> > > > parties.
> > > >
> > > > 1.  Different language implementations within the Arrow project that
> > want
> > > > to call into each other's code.  We still don't have a great story
> > around
> > > > this in terms of reusable libraries and questions like [1] are a
> > > motivating
> > > > examples of making something better in this context.
> > > > 2.  third-parties wishing to support/integrate with Arrow.  Some
> > > > conjectures about these users:
> > > >   - Users in this group are NOT necessarily familiar with existing
> > > > technologies Arrow uses (i.e. flatbuffers)
> > > >   - The stability of the API is the primary concern (consumers don't
> > want
> > > > to change when a new version of the library ships)
> > > >   - An important secondary concern is additional libraries that need to
> > > be
> > > > integrated in addition to the API
> > > >
> > > > The main debate points seems to be:
> > > >
> > > > 1.  Vector/Array oriented API vs existing Record Batch.  Will an
> > > additional
> > > > column oriented API become too much of a maintenance headache/cause
> > > > fragmentation?
> > > >
> > > >  - In my mind the question here is which set of users we are
> > > prioritizing.
> > > > IMO the combination of flatbuffers and translation to/from RecordBatch
> > > > format offers too much friction to make it easy for a third-party
> > > > implementer to use. If we are prioritizing for our own internal
> > > use-cases I
> > > > think we should try out a RecordBatch+Flatbuffers based C-API. We
> > already
> > > > have all the necessary building blocks.
> > > >
> > >
> > > If a C function passes you a string containing a RecordBatch
> > > Flatbuffers message, what happens next? This message has to be
> > > reassembled into a recursive data structure before you can "do"
> > > anything with it. Are we expecting every third party project to
> > > implement:
> > >
> > > A. Data structures appropriate to represent a logical "field" in a
> > > record batch (which have to be recursive to account for nested types'
> > > children)
> > > B. The logic to convert from the flattened Flatbuffers representation
> > > to some implementation of A
> > >
> > > I'm arguing that we should provide both to third parties. To build B,
> > > you need A. Some consumers will only use A. This discussion is
> > > essentially about developing an ultraminimalist "drop-in" C
> > > implementation of A.
> > >
> > > > 2.  How onerous is the dependency on flat-buffers both from a learning
> > > > curve perspective and as dependency for third-party integrators?
> > > > - Flatbuffers aren't entirely straight-forward and I think if we do
> > move
> > > > forward with an API based on Column/Array we should consider
> > alternatives
> > > > as long as the necessary parsing code can be done in a small amount of
> > > code
> > > > (I'm personally against JSON for this, but can see the arguments for
> > it).
> > > >
> > > > 3.  Do all existing library implementations need to support both
> > > > Column/Array a ABI?  How will compliance be checked for the new
> > API/ABI?
> > > >
> > > > - I'm still thinking this through.
> > > >
> > > > [1]
> > > >
> > >
> > https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
> > > >
> > > > On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org>
> > > wrote:
> > > >
> > > > > I'd like to hear more opinions from others on this topic. This
> > > conversation
> > > > > seems mostly dominated by comments from myself, Wes and Antoine.
> > > > >
> > > > > I think it is reasonable to argue that keeping any ABI (or
> > > header/struct
> > > > > pattern) as narrow as possible would allow us to minimize overlap
> > with
> > > the
> > > > > existing in-memory specification. In Arrow's case, this could be as
> > > simple
> > > > > as a single memory pointer for schema (backed by flatbuffers) and a
> > > single
> > > > > memory location for data (that references the record batch header,
> > > which in
> > > > > turn provides pointers into the actual arrow data). Extensions would
> > > need
> > > > > to be added for reference management as done here but I continue to
> > > think
> > > > > we should defer discussion of that until the base data structures are
> > > > > resolved. I see the comments here as arguing for a much broader ABI,
> > in
> > > > > part to support having people build "Arrow" components that
> > > interconnect
> > > > > using this new interface. I understand the desire to expand the ABI
> > to
> > > be
> > > > > driven by needs to reduce dependencies and ease usability.
> > > > >
> > > > > The representation within the related patch is being presented as a
> > > way for
> > > > > applications to share Arrow data but is not easily accessible to all
> > > > > languages. I want to avoid a situation where someone says "I produced
> > > an
> > > > > Arrow API" when what they've really done is created a C interface
> > which
> > > > > only a small subset of languages can actually leverage. For example,
> > > every
> > > > > language now knows how to parse the existing schema definition as
> > > rendered
> > > > > in flatbuf. In order to interact with something that implements this
> > > new
> > > > > pattern one would also be required to implement completely new schema
> > > > > consumption code. In the proposal itself it suggests this (for
> > example
> > > > > enhancing the C++ library to consume structures produced this way).
> > > > >
> > > > > As I said, I really want to hear more opinions. Running this past
> > > various
> > > > > developers I know, many have echoed my concerns but that really
> > doesn't
> > > > > matter (and who knows how much of that is colored by my presentation
> > > of the
> > > > > issue). What do people here think? If someone builds an "Arrow"
> > library
> > > > > that implements this set of structures, how does one use it in Node?
> > In
> > > > > Java? Does it drive creation of a secondary set of interfaces in each
> > > of
> > > > > those languages to work with this kind of pattern? (For example, in a
> > > JVM
> > > > > view of the world, working with a plain struct in java rather than a
> > > set of
> > > > > memory pointers against our existing IPC formats would be quite
> > > painful and
> > > > > we'd definitely need to create some glue code for users. I worry the
> > > same
> > > > > pattern would occur in many other languages.)
> > > > >
> > > > > To respond directly to some of Wes's most recent comments from the
> > > email
> > > > > below. I struggle to map your description of the situation to the
> > rest
> > > of
> > > > > the thread and the proposed patch.  For example, you say that a
> > > non-goal is
> > > > > "creating a new canonical way to serialize metadata" bute the patch
> > > > > proposes a concrete string based encoding system to describe data
> > > types.
> > > > > Aren't those things in conflict?
> > > > >
> > > > > I'll also think more on this and challenge my own perspective. This
> > > isn't
> > > > > where my focus is so my comments aren't as developed/thoughtful as
> > I'd
> > > > > like.
> > > > >
> > > > >
> > > > > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi Jacques,
> > > > > >
> > > > > > I think we've veered off course a bit and maybe we could reframe
> > the
> > > > > > discussion.
> > > > > >
> > > > > > Goals
> > > > > > * A "drop-in" header-only C file that projects can use as a
> > > > > > programming interface either internally only or to expose in-memory
> > > > > > data structures between C functions at call sites. Ideally little
> > to
> > > > > > no disassembly/reassembly should be required on either "side" of
> > the
> > > > > > call site.
> > > > > > * Simplifying adoption of Arrow for C programmers, or languages
> > based
> > > > > > around C FFI
> > > > > >
> > > > > > Non-goals
> > > > > > * Expanding the columnar format or creating an alternative
> > canonical
> > > > > > in-memory representation
> > > > > > * Creating a new canonical way to serialize metadata
> > > > > >
> > > > > > Note that this use case has been on my mind for more than 2 years:
> > > > > > https://issues.apache.org/jira/browse/ARROW-1058
> > > > > >
> > > > > > I think there are a couple of potentially misleading things at play
> > > here
> > > > > >
> > > > > > 1. The use of the word "protocol". In C, a struct has a
> > well-defined
> > > > > > binary layout, so a C API is also an ABI. Using C structs to
> > > > > > communicate data can be considered to be a protocol, but it means
> > > > > > something different in the context of the "Arrow protocol". I think
> > > we
> > > > > > need to call this a "C API"
> > > > > >
> > > > > > 2. The documentation for this in Antoine's PR is in the format/
> > > > > > directory. It would probably be better to have a "C API" section in
> > > > > > the documentation.
> > > > > >
> > > > > > The header file under discussion and the documentation about it is
> > > > > > best considered as a "library".
> > > > > >
> > > > > > It might be useful at some point to create a C99 implementation of
> > > the
> > > > > > IPC protocol as well using FlatCC with the goal of having a
> > complete
> > > > > > implementation of the columnar format in C with minimal binary
> > > > > > footprint. This is analogous to the NanoPB project which is an
> > > > > > implementation of Protocol Buffers with small code size
> > > > > >
> > > > > > https://github.com/nanopb/nanopb
> > > > > >
> > > > > > Let me know if this makes more sense.
> > > > > >
> > > > > > I think it's important to communicate clearly about this primarily
> > > for
> > > > > > the benefit of the outside world which can confuse easily as we
> > have
> > > > > > observed over the last few years =)
> > > > > >
> > > > > > Wes
> > > > > >
> > > > > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > I disagree with this statement:
> > > > > > >
> > > > > > > - the IPC format is meant for serialization while the C data
> > > protocol
> > > > > is
> > > > > > > meants for in-memory communication, so different concerns apply
> > > > > > >
> > > > > > > If that is how the a particular implementation presents it, that
> > > is a
> > > > > > > weaknesses of the implementation, not the format. The primary use
> > > case
> > > > > I
> > > > > > > was focused on when working on the initial format was
> > communication
> > > > > > within
> > > > > > > the same process. It seems like this is being used as a basis for
> > > the
> > > > > > > introduction of new things when the premise is inconsistent with
> > > the
> > > > > > > intention of the creation. The specific reason we used
> > flatbuffers
> > > in
> > > > > the
> > > > > > > project was to collapse the separation of in-process and
> > > out-of-process
> > > > > > > communication. It means the same thing it does with the Arrow
> > data
> > > > > > itself:
> > > > > > > that a consumer doesn't have to use a particular library to
> > > interact
> > > > > with
> > > > > > > and use the data.
> > > > > > >
> > > > > > > It seems like there are two ideas here:
> > > > > > >
> > > > > > > 1) How do we make it easier for people to use Arrow?
> > > > > > > 2) Should we implement a new in memory representation of Arrow
> > > that is
> > > > > > > language specific.
> > > > > > >
> > > > > > > I'm entirely in support of number one. If for a particular type
> > of
> > > > > > domain,
> > > > > > > people want an easier way to interact with Arrow, let's make a
> > new
> > > > > > library
> > > > > > > that helps with that. In easy of our current libraries, we do
> > many
> > > > > things
> > > > > > > to make it easier to work with Arrow. None of those require a
> > > change to
> > > > > > the
> > > > > > > core format or are formalized as a new in-memory standard. The
> > > > > in-memory
> > > > > > > representation of rust or javascript or java objects are
> > > implementation
> > > > > > > details.
> > > > > > >
> > > > > > > I'm against number two as it creates a fragmentation problem.
> > > Arrow is
> > > > > > > about having a single canonical format for memory for both
> > > metadata and
> > > > > > > data. Having multiple in-memory formats (especially when some are
> > > not
> > > > > > > language independent) is counter to the goals of the project.
> > > > > >
> > > > > > I don't think anyone is proposing anything that would cause
> > > > > fragmentation.
> > > > > >
> > > > > > A central question is whether it is useful to define a reusable C
> > ABI
> > > > > > for the Arrow columnar format, and if there is sufficient
> > interest, a
> > > > > > tiny C implementation of the IPC protocol (which uses the
> > Flatbuffers
> > > > > > message) that assembles and disassembles the data structures
> > defined
> > > > > > in the C ABI.
> > > > > >
> > > > > > We could separately create a tiny implementation of the Arrow IPC
> > > > > > protocol using FlatCC that could be dropped into applications
> > > > > > requiring only a C compiler and nothing else.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Two other, separate comments:
> > > > > > > 1) I don't understand the idea that we need to change the way
> > Arrow
> > > > > > > fundamentally works so that people can avoid using a dependency.
> > > If the
> > > > > > > dependency is small, open source and easy to build, people can
> > > fork it
> > > > > > and
> > > > > > > include directly if they want to. Let's not violate project
> > > principles
> > > > > > > because DuckDB has a religious perspective on dependencies. If
> > the
> > > > > > problem
> > > > > > > is people have to swallow too large of a pill to do basic things
> > > with
> > > > > > Arrow
> > > > > > > in C, let's focus on fixing that (to our definition of ease, not
> > > > > someone
> > > > > > > else's). If FlatCC solves some those things, great. If we need to
> > > > > build a
> > > > > > > baby integration library that is more C centric, great. Neither
> > of
> > > > > those
> > > > > > > things require implementing something at the format level.
> > > > > > >
> > > > > > > 2) It seems like we should discuss the data structure problem
> > > > > separately
> > > > > > > from the reference management concern.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <wesmckinn@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > hi Antoine,
> > > > > > > >
> > > > > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <
> > > antoine@python.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > > > > > A couple things:
> > > > > > > > > >
> > > > > > > > > > * I think a C protocol / FFI for Arrow array/vectors would
> > be
> > > > > > better
> > > > > > > > > > to have the same "shape" as an assembled array. Note that
> > > the C
> > > > > > > > > > structs here have very nearly the same "shape" as the data
> > > > > > structure
> > > > > > > > > > representing a C++ Array object [1]. The disassembly and
> > > > > reassembly
> > > > > > > > > > here is substantially simpler than the IPC protocol. A
> > > recursive
> > > > > > > > > > structure in Flatbuffers would make RecordBatch messages
> > much
> > > > > > larger,
> > > > > > > > > > so the flattened / disassembled representation we use for
> > > > > > serialized
> > > > > > > > > > record batches is the correct one
> > > > > > > > >
> > > > > > > > > I'm not sure I agree:
> > > > > > > > >
> > > > > > > > > - indeed, it's not a coincidence that the ArrowArray struct
> > > looks
> > > > > > quite
> > > > > > > > > closely like the C++ ArrayData object :-)  We have good
> > > experience
> > > > > > with
> > > > > > > > > that abstraction and it has proven to work quite well
> > > > > > > > >
> > > > > > > > > - the IPC format is meant for serialization while the C data
> > > > > > protocol is
> > > > > > > > > meants for in-memory communication, so different concerns
> > apply
> > > > > > > > >
> > > > > > > > > - the fact that this makes the layout slightly larger doesn't
> > > seem
> > > > > > > > > important at all; we're not talking about transferring data
> > > over
> > > > > the
> > > > > > wire
> > > > > > > > >
> > > > > > > > > There's also another argument for having a recursive struct:
> > it
> > > > > > > > > simplifies how the data type is represented, since we can
> > > encode
> > > > > each
> > > > > > > > > child type individually instead of encoding it in the
> > parent's
> > > > > format
> > > > > > > > > string (same applies for metadata and individual flags).
> > > > > > > > >
> > > > > > > >
> > > > > > > > I was saying something different here. I was making an argument
> > > about
> > > > > > > > why we use the flattened array-of-structs in the IPC protocol.
> > > One
> > > > > > > > reason is that it's a more compact representation. That is not
> > > very
> > > > > > > > important here because this protocol is only for *in-process*
> > > (for
> > > > > > > > languages that have a C FFI facility) rather than
> > *inter-process*
> > > > > > > > communication.
> > > > > > > >
> > > > > > > > I agree also that the type encoding is simple, here, too, since
> > > we
> > > > > > > > aren't having to split the schema and record batch between
> > > different
> > > > > > > > serialized messages. There is some potential waste with having
> > to
> > > > > > > > populate the type fields multiple times when communicating a
> > > sequence
> > > > > > > > of "chunks" from the same logical dataset.
> > > > > > > >
> > > > > > > > > > * The "formal" C protocol having the "assembled" shape
> > means
> > > that
> > > > > > many
> > > > > > > > > > minimal Arrow users won't have to implement any separate
> > data
> > > > > > > > > > structures. They can just use the C struct directly or a
> > > slightly
> > > > > > > > > > wrapped version thereof with some convenience functions.
> > > > > > > > >
> > > > > > > > > Yes, but the same applies to the current proposal.
> > > > > > > > >
> > > > > > > > > > * I think that requiring building a Flatbuffer for minimal
> > > use
> > > > > > cases
> > > > > > > > > > (e.g. communicating simple record batches with primitive
> > > types)
> > > > > > passes
> > > > > > > > > > on implementation burden to minimal users.
> > > > > > > > >
> > > > > > > > > It certainly does.
> > > > > > > > >
> > > > > > > > > > I think the mantra of the C protocol should be the
> > following:
> > > > > > > > > >
> > > > > > > > > > * Users of the protocol have to write little to no code to
> > > use
> > > > > it.
> > > > > > For
> > > > > > > > > > example, populating an INT32 array should require only a
> > few
> > > > > lines
> > > > > > of
> > > > > > > > > > code
> > > > > > > > >
> > > > > > > > > Agreed.  As a sidenote, the spec should have an example of
> > > doing
> > > > > > this in
> > > > > > > > > raw C.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > Antoine.
> > > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

I removing all my objections to this work.

I wish there was more feedback from additional community members. I
continue to be concerned about fragmentation. I don't agree with the
arguments here that we need to add a new api to make it easy for people to
*not* use Arrow codebase. It seems like a punt on building useful libraries
within the project that will ultimately hurt the interoperability story.

As a side note, it seems like much of this is about people's distaste for
flatbuffers. I know I regret using it. If we had a chance to do it over
again, I would have chosen to use protobuf for everything except the data
header, where I would hand write the encoding (since it is so simple
anyway). If it is such a problem that people are contorting to work around
it, maybe we should address that? Just a thought.

Thanks for the discourse and patience.

On Wed, Oct 2, 2019 at 10:12 PM Micah Kornfield <em...@gmail.com>
wrote:

> Hi Wes,
> I agree for third-parties "A" (Field data structures) is the most useful.
>
> At least in my mind the discussion was for both first and third-parties.  I
> was trying to point out that "A" is less necessary as a first step for
> first-party integrations and could potentially require more effort if we
> already have the code that does "B" (field reassembly).
>
> Thanks,
> Micah
>
> On Wed, Oct 2, 2019 at 10:28 PM Wes McKinney <we...@gmail.com> wrote:
>
> > On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> > >
> > > I've tried to summarize my understanding of the debate so far and give
> > some
> > > initial thoughts. I think there are two potentially different sets of
> > users
> > > that we are targeting with a stable C API/ABI ourselves and external
> > > parties.
> > >
> > > 1.  Different language implementations within the Arrow project that
> want
> > > to call into each other's code.  We still don't have a great story
> around
> > > this in terms of reusable libraries and questions like [1] are a
> > motivating
> > > examples of making something better in this context.
> > > 2.  third-parties wishing to support/integrate with Arrow.  Some
> > > conjectures about these users:
> > >   - Users in this group are NOT necessarily familiar with existing
> > > technologies Arrow uses (i.e. flatbuffers)
> > >   - The stability of the API is the primary concern (consumers don't
> want
> > > to change when a new version of the library ships)
> > >   - An important secondary concern is additional libraries that need to
> > be
> > > integrated in addition to the API
> > >
> > > The main debate points seems to be:
> > >
> > > 1.  Vector/Array oriented API vs existing Record Batch.  Will an
> > additional
> > > column oriented API become too much of a maintenance headache/cause
> > > fragmentation?
> > >
> > >  - In my mind the question here is which set of users we are
> > prioritizing.
> > > IMO the combination of flatbuffers and translation to/from RecordBatch
> > > format offers too much friction to make it easy for a third-party
> > > implementer to use. If we are prioritizing for our own internal
> > use-cases I
> > > think we should try out a RecordBatch+Flatbuffers based C-API. We
> already
> > > have all the necessary building blocks.
> > >
> >
> > If a C function passes you a string containing a RecordBatch
> > Flatbuffers message, what happens next? This message has to be
> > reassembled into a recursive data structure before you can "do"
> > anything with it. Are we expecting every third party project to
> > implement:
> >
> > A. Data structures appropriate to represent a logical "field" in a
> > record batch (which have to be recursive to account for nested types'
> > children)
> > B. The logic to convert from the flattened Flatbuffers representation
> > to some implementation of A
> >
> > I'm arguing that we should provide both to third parties. To build B,
> > you need A. Some consumers will only use A. This discussion is
> > essentially about developing an ultraminimalist "drop-in" C
> > implementation of A.
> >
> > > 2.  How onerous is the dependency on flat-buffers both from a learning
> > > curve perspective and as dependency for third-party integrators?
> > > - Flatbuffers aren't entirely straight-forward and I think if we do
> move
> > > forward with an API based on Column/Array we should consider
> alternatives
> > > as long as the necessary parsing code can be done in a small amount of
> > code
> > > (I'm personally against JSON for this, but can see the arguments for
> it).
> > >
> > > 3.  Do all existing library implementations need to support both
> > > Column/Array a ABI?  How will compliance be checked for the new
> API/ABI?
> > >
> > > - I'm still thinking this through.
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
> > >
> > > On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org>
> > wrote:
> > >
> > > > I'd like to hear more opinions from others on this topic. This
> > conversation
> > > > seems mostly dominated by comments from myself, Wes and Antoine.
> > > >
> > > > I think it is reasonable to argue that keeping any ABI (or
> > header/struct
> > > > pattern) as narrow as possible would allow us to minimize overlap
> with
> > the
> > > > existing in-memory specification. In Arrow's case, this could be as
> > simple
> > > > as a single memory pointer for schema (backed by flatbuffers) and a
> > single
> > > > memory location for data (that references the record batch header,
> > which in
> > > > turn provides pointers into the actual arrow data). Extensions would
> > need
> > > > to be added for reference management as done here but I continue to
> > think
> > > > we should defer discussion of that until the base data structures are
> > > > resolved. I see the comments here as arguing for a much broader ABI,
> in
> > > > part to support having people build "Arrow" components that
> > interconnect
> > > > using this new interface. I understand the desire to expand the ABI
> to
> > be
> > > > driven by needs to reduce dependencies and ease usability.
> > > >
> > > > The representation within the related patch is being presented as a
> > way for
> > > > applications to share Arrow data but is not easily accessible to all
> > > > languages. I want to avoid a situation where someone says "I produced
> > an
> > > > Arrow API" when what they've really done is created a C interface
> which
> > > > only a small subset of languages can actually leverage. For example,
> > every
> > > > language now knows how to parse the existing schema definition as
> > rendered
> > > > in flatbuf. In order to interact with something that implements this
> > new
> > > > pattern one would also be required to implement completely new schema
> > > > consumption code. In the proposal itself it suggests this (for
> example
> > > > enhancing the C++ library to consume structures produced this way).
> > > >
> > > > As I said, I really want to hear more opinions. Running this past
> > various
> > > > developers I know, many have echoed my concerns but that really
> doesn't
> > > > matter (and who knows how much of that is colored by my presentation
> > of the
> > > > issue). What do people here think? If someone builds an "Arrow"
> library
> > > > that implements this set of structures, how does one use it in Node?
> In
> > > > Java? Does it drive creation of a secondary set of interfaces in each
> > of
> > > > those languages to work with this kind of pattern? (For example, in a
> > JVM
> > > > view of the world, working with a plain struct in java rather than a
> > set of
> > > > memory pointers against our existing IPC formats would be quite
> > painful and
> > > > we'd definitely need to create some glue code for users. I worry the
> > same
> > > > pattern would occur in many other languages.)
> > > >
> > > > To respond directly to some of Wes's most recent comments from the
> > email
> > > > below. I struggle to map your description of the situation to the
> rest
> > of
> > > > the thread and the proposed patch.  For example, you say that a
> > non-goal is
> > > > "creating a new canonical way to serialize metadata" bute the patch
> > > > proposes a concrete string based encoding system to describe data
> > types.
> > > > Aren't those things in conflict?
> > > >
> > > > I'll also think more on this and challenge my own perspective. This
> > isn't
> > > > where my focus is so my comments aren't as developed/thoughtful as
> I'd
> > > > like.
> > > >
> > > >
> > > > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > hi Jacques,
> > > > >
> > > > > I think we've veered off course a bit and maybe we could reframe
> the
> > > > > discussion.
> > > > >
> > > > > Goals
> > > > > * A "drop-in" header-only C file that projects can use as a
> > > > > programming interface either internally only or to expose in-memory
> > > > > data structures between C functions at call sites. Ideally little
> to
> > > > > no disassembly/reassembly should be required on either "side" of
> the
> > > > > call site.
> > > > > * Simplifying adoption of Arrow for C programmers, or languages
> based
> > > > > around C FFI
> > > > >
> > > > > Non-goals
> > > > > * Expanding the columnar format or creating an alternative
> canonical
> > > > > in-memory representation
> > > > > * Creating a new canonical way to serialize metadata
> > > > >
> > > > > Note that this use case has been on my mind for more than 2 years:
> > > > > https://issues.apache.org/jira/browse/ARROW-1058
> > > > >
> > > > > I think there are a couple of potentially misleading things at play
> > here
> > > > >
> > > > > 1. The use of the word "protocol". In C, a struct has a
> well-defined
> > > > > binary layout, so a C API is also an ABI. Using C structs to
> > > > > communicate data can be considered to be a protocol, but it means
> > > > > something different in the context of the "Arrow protocol". I think
> > we
> > > > > need to call this a "C API"
> > > > >
> > > > > 2. The documentation for this in Antoine's PR is in the format/
> > > > > directory. It would probably be better to have a "C API" section in
> > > > > the documentation.
> > > > >
> > > > > The header file under discussion and the documentation about it is
> > > > > best considered as a "library".
> > > > >
> > > > > It might be useful at some point to create a C99 implementation of
> > the
> > > > > IPC protocol as well using FlatCC with the goal of having a
> complete
> > > > > implementation of the columnar format in C with minimal binary
> > > > > footprint. This is analogous to the NanoPB project which is an
> > > > > implementation of Protocol Buffers with small code size
> > > > >
> > > > > https://github.com/nanopb/nanopb
> > > > >
> > > > > Let me know if this makes more sense.
> > > > >
> > > > > I think it's important to communicate clearly about this primarily
> > for
> > > > > the benefit of the outside world which can confuse easily as we
> have
> > > > > observed over the last few years =)
> > > > >
> > > > > Wes
> > > > >
> > > > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > I disagree with this statement:
> > > > > >
> > > > > > - the IPC format is meant for serialization while the C data
> > protocol
> > > > is
> > > > > > meants for in-memory communication, so different concerns apply
> > > > > >
> > > > > > If that is how the a particular implementation presents it, that
> > is a
> > > > > > weaknesses of the implementation, not the format. The primary use
> > case
> > > > I
> > > > > > was focused on when working on the initial format was
> communication
> > > > > within
> > > > > > the same process. It seems like this is being used as a basis for
> > the
> > > > > > introduction of new things when the premise is inconsistent with
> > the
> > > > > > intention of the creation. The specific reason we used
> flatbuffers
> > in
> > > > the
> > > > > > project was to collapse the separation of in-process and
> > out-of-process
> > > > > > communication. It means the same thing it does with the Arrow
> data
> > > > > itself:
> > > > > > that a consumer doesn't have to use a particular library to
> > interact
> > > > with
> > > > > > and use the data.
> > > > > >
> > > > > > It seems like there are two ideas here:
> > > > > >
> > > > > > 1) How do we make it easier for people to use Arrow?
> > > > > > 2) Should we implement a new in memory representation of Arrow
> > that is
> > > > > > language specific.
> > > > > >
> > > > > > I'm entirely in support of number one. If for a particular type
> of
> > > > > domain,
> > > > > > people want an easier way to interact with Arrow, let's make a
> new
> > > > > library
> > > > > > that helps with that. In easy of our current libraries, we do
> many
> > > > things
> > > > > > to make it easier to work with Arrow. None of those require a
> > change to
> > > > > the
> > > > > > core format or are formalized as a new in-memory standard. The
> > > > in-memory
> > > > > > representation of rust or javascript or java objects are
> > implementation
> > > > > > details.
> > > > > >
> > > > > > I'm against number two as it creates a fragmentation problem.
> > Arrow is
> > > > > > about having a single canonical format for memory for both
> > metadata and
> > > > > > data. Having multiple in-memory formats (especially when some are
> > not
> > > > > > language independent) is counter to the goals of the project.
> > > > >
> > > > > I don't think anyone is proposing anything that would cause
> > > > fragmentation.
> > > > >
> > > > > A central question is whether it is useful to define a reusable C
> ABI
> > > > > for the Arrow columnar format, and if there is sufficient
> interest, a
> > > > > tiny C implementation of the IPC protocol (which uses the
> Flatbuffers
> > > > > message) that assembles and disassembles the data structures
> defined
> > > > > in the C ABI.
> > > > >
> > > > > We could separately create a tiny implementation of the Arrow IPC
> > > > > protocol using FlatCC that could be dropped into applications
> > > > > requiring only a C compiler and nothing else.
> > > > >
> > > > >
> > > > > >
> > > > > > Two other, separate comments:
> > > > > > 1) I don't understand the idea that we need to change the way
> Arrow
> > > > > > fundamentally works so that people can avoid using a dependency.
> > If the
> > > > > > dependency is small, open source and easy to build, people can
> > fork it
> > > > > and
> > > > > > include directly if they want to. Let's not violate project
> > principles
> > > > > > because DuckDB has a religious perspective on dependencies. If
> the
> > > > > problem
> > > > > > is people have to swallow too large of a pill to do basic things
> > with
> > > > > Arrow
> > > > > > in C, let's focus on fixing that (to our definition of ease, not
> > > > someone
> > > > > > else's). If FlatCC solves some those things, great. If we need to
> > > > build a
> > > > > > baby integration library that is more C centric, great. Neither
> of
> > > > those
> > > > > > things require implementing something at the format level.
> > > > > >
> > > > > > 2) It seems like we should discuss the data structure problem
> > > > separately
> > > > > > from the reference management concern.
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <wesmckinn@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > hi Antoine,
> > > > > > >
> > > > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <
> > antoine@python.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > > > > A couple things:
> > > > > > > > >
> > > > > > > > > * I think a C protocol / FFI for Arrow array/vectors would
> be
> > > > > better
> > > > > > > > > to have the same "shape" as an assembled array. Note that
> > the C
> > > > > > > > > structs here have very nearly the same "shape" as the data
> > > > > structure
> > > > > > > > > representing a C++ Array object [1]. The disassembly and
> > > > reassembly
> > > > > > > > > here is substantially simpler than the IPC protocol. A
> > recursive
> > > > > > > > > structure in Flatbuffers would make RecordBatch messages
> much
> > > > > larger,
> > > > > > > > > so the flattened / disassembled representation we use for
> > > > > serialized
> > > > > > > > > record batches is the correct one
> > > > > > > >
> > > > > > > > I'm not sure I agree:
> > > > > > > >
> > > > > > > > - indeed, it's not a coincidence that the ArrowArray struct
> > looks
> > > > > quite
> > > > > > > > closely like the C++ ArrayData object :-)  We have good
> > experience
> > > > > with
> > > > > > > > that abstraction and it has proven to work quite well
> > > > > > > >
> > > > > > > > - the IPC format is meant for serialization while the C data
> > > > > protocol is
> > > > > > > > meants for in-memory communication, so different concerns
> apply
> > > > > > > >
> > > > > > > > - the fact that this makes the layout slightly larger doesn't
> > seem
> > > > > > > > important at all; we're not talking about transferring data
> > over
> > > > the
> > > > > wire
> > > > > > > >
> > > > > > > > There's also another argument for having a recursive struct:
> it
> > > > > > > > simplifies how the data type is represented, since we can
> > encode
> > > > each
> > > > > > > > child type individually instead of encoding it in the
> parent's
> > > > format
> > > > > > > > string (same applies for metadata and individual flags).
> > > > > > > >
> > > > > > >
> > > > > > > I was saying something different here. I was making an argument
> > about
> > > > > > > why we use the flattened array-of-structs in the IPC protocol.
> > One
> > > > > > > reason is that it's a more compact representation. That is not
> > very
> > > > > > > important here because this protocol is only for *in-process*
> > (for
> > > > > > > languages that have a C FFI facility) rather than
> *inter-process*
> > > > > > > communication.
> > > > > > >
> > > > > > > I agree also that the type encoding is simple, here, too, since
> > we
> > > > > > > aren't having to split the schema and record batch between
> > different
> > > > > > > serialized messages. There is some potential waste with having
> to
> > > > > > > populate the type fields multiple times when communicating a
> > sequence
> > > > > > > of "chunks" from the same logical dataset.
> > > > > > >
> > > > > > > > > * The "formal" C protocol having the "assembled" shape
> means
> > that
> > > > > many
> > > > > > > > > minimal Arrow users won't have to implement any separate
> data
> > > > > > > > > structures. They can just use the C struct directly or a
> > slightly
> > > > > > > > > wrapped version thereof with some convenience functions.
> > > > > > > >
> > > > > > > > Yes, but the same applies to the current proposal.
> > > > > > > >
> > > > > > > > > * I think that requiring building a Flatbuffer for minimal
> > use
> > > > > cases
> > > > > > > > > (e.g. communicating simple record batches with primitive
> > types)
> > > > > passes
> > > > > > > > > on implementation burden to minimal users.
> > > > > > > >
> > > > > > > > It certainly does.
> > > > > > > >
> > > > > > > > > I think the mantra of the C protocol should be the
> following:
> > > > > > > > >
> > > > > > > > > * Users of the protocol have to write little to no code to
> > use
> > > > it.
> > > > > For
> > > > > > > > > example, populating an INT32 array should require only a
> few
> > > > lines
> > > > > of
> > > > > > > > > code
> > > > > > > >
> > > > > > > > Agreed.  As a sidenote, the spec should have an example of
> > doing
> > > > > this in
> > > > > > > > raw C.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > Antoine.
> > > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Micah Kornfield <em...@gmail.com>]

Hi Wes,
I agree for third-parties "A" (Field data structures) is the most useful.

At least in my mind the discussion was for both first and third-parties.  I
was trying to point out that "A" is less necessary as a first step for
first-party integrations and could potentially require more effort if we
already have the code that does "B" (field reassembly).

Thanks,
Micah

On Wed, Oct 2, 2019 at 10:28 PM Wes McKinney <we...@gmail.com> wrote:

> On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > I've tried to summarize my understanding of the debate so far and give
> some
> > initial thoughts. I think there are two potentially different sets of
> users
> > that we are targeting with a stable C API/ABI ourselves and external
> > parties.
> >
> > 1.  Different language implementations within the Arrow project that want
> > to call into each other's code.  We still don't have a great story around
> > this in terms of reusable libraries and questions like [1] are a
> motivating
> > examples of making something better in this context.
> > 2.  third-parties wishing to support/integrate with Arrow.  Some
> > conjectures about these users:
> >   - Users in this group are NOT necessarily familiar with existing
> > technologies Arrow uses (i.e. flatbuffers)
> >   - The stability of the API is the primary concern (consumers don't want
> > to change when a new version of the library ships)
> >   - An important secondary concern is additional libraries that need to
> be
> > integrated in addition to the API
> >
> > The main debate points seems to be:
> >
> > 1.  Vector/Array oriented API vs existing Record Batch.  Will an
> additional
> > column oriented API become too much of a maintenance headache/cause
> > fragmentation?
> >
> >  - In my mind the question here is which set of users we are
> prioritizing.
> > IMO the combination of flatbuffers and translation to/from RecordBatch
> > format offers too much friction to make it easy for a third-party
> > implementer to use. If we are prioritizing for our own internal
> use-cases I
> > think we should try out a RecordBatch+Flatbuffers based C-API. We already
> > have all the necessary building blocks.
> >
>
> If a C function passes you a string containing a RecordBatch
> Flatbuffers message, what happens next? This message has to be
> reassembled into a recursive data structure before you can "do"
> anything with it. Are we expecting every third party project to
> implement:
>
> A. Data structures appropriate to represent a logical "field" in a
> record batch (which have to be recursive to account for nested types'
> children)
> B. The logic to convert from the flattened Flatbuffers representation
> to some implementation of A
>
> I'm arguing that we should provide both to third parties. To build B,
> you need A. Some consumers will only use A. This discussion is
> essentially about developing an ultraminimalist "drop-in" C
> implementation of A.
>
> > 2.  How onerous is the dependency on flat-buffers both from a learning
> > curve perspective and as dependency for third-party integrators?
> > - Flatbuffers aren't entirely straight-forward and I think if we do move
> > forward with an API based on Column/Array we should consider alternatives
> > as long as the necessary parsing code can be done in a small amount of
> code
> > (I'm personally against JSON for this, but can see the arguments for it).
> >
> > 3.  Do all existing library implementations need to support both
> > Column/Array a ABI?  How will compliance be checked for the new API/ABI?
> >
> > - I'm still thinking this through.
> >
> > [1]
> >
> https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
> >
> > On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org>
> wrote:
> >
> > > I'd like to hear more opinions from others on this topic. This
> conversation
> > > seems mostly dominated by comments from myself, Wes and Antoine.
> > >
> > > I think it is reasonable to argue that keeping any ABI (or
> header/struct
> > > pattern) as narrow as possible would allow us to minimize overlap with
> the
> > > existing in-memory specification. In Arrow's case, this could be as
> simple
> > > as a single memory pointer for schema (backed by flatbuffers) and a
> single
> > > memory location for data (that references the record batch header,
> which in
> > > turn provides pointers into the actual arrow data). Extensions would
> need
> > > to be added for reference management as done here but I continue to
> think
> > > we should defer discussion of that until the base data structures are
> > > resolved. I see the comments here as arguing for a much broader ABI, in
> > > part to support having people build "Arrow" components that
> interconnect
> > > using this new interface. I understand the desire to expand the ABI to
> be
> > > driven by needs to reduce dependencies and ease usability.
> > >
> > > The representation within the related patch is being presented as a
> way for
> > > applications to share Arrow data but is not easily accessible to all
> > > languages. I want to avoid a situation where someone says "I produced
> an
> > > Arrow API" when what they've really done is created a C interface which
> > > only a small subset of languages can actually leverage. For example,
> every
> > > language now knows how to parse the existing schema definition as
> rendered
> > > in flatbuf. In order to interact with something that implements this
> new
> > > pattern one would also be required to implement completely new schema
> > > consumption code. In the proposal itself it suggests this (for example
> > > enhancing the C++ library to consume structures produced this way).
> > >
> > > As I said, I really want to hear more opinions. Running this past
> various
> > > developers I know, many have echoed my concerns but that really doesn't
> > > matter (and who knows how much of that is colored by my presentation
> of the
> > > issue). What do people here think? If someone builds an "Arrow" library
> > > that implements this set of structures, how does one use it in Node? In
> > > Java? Does it drive creation of a secondary set of interfaces in each
> of
> > > those languages to work with this kind of pattern? (For example, in a
> JVM
> > > view of the world, working with a plain struct in java rather than a
> set of
> > > memory pointers against our existing IPC formats would be quite
> painful and
> > > we'd definitely need to create some glue code for users. I worry the
> same
> > > pattern would occur in many other languages.)
> > >
> > > To respond directly to some of Wes's most recent comments from the
> email
> > > below. I struggle to map your description of the situation to the rest
> of
> > > the thread and the proposed patch.  For example, you say that a
> non-goal is
> > > "creating a new canonical way to serialize metadata" bute the patch
> > > proposes a concrete string based encoding system to describe data
> types.
> > > Aren't those things in conflict?
> > >
> > > I'll also think more on this and challenge my own perspective. This
> isn't
> > > where my focus is so my comments aren't as developed/thoughtful as I'd
> > > like.
> > >
> > >
> > > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > hi Jacques,
> > > >
> > > > I think we've veered off course a bit and maybe we could reframe the
> > > > discussion.
> > > >
> > > > Goals
> > > > * A "drop-in" header-only C file that projects can use as a
> > > > programming interface either internally only or to expose in-memory
> > > > data structures between C functions at call sites. Ideally little to
> > > > no disassembly/reassembly should be required on either "side" of the
> > > > call site.
> > > > * Simplifying adoption of Arrow for C programmers, or languages based
> > > > around C FFI
> > > >
> > > > Non-goals
> > > > * Expanding the columnar format or creating an alternative canonical
> > > > in-memory representation
> > > > * Creating a new canonical way to serialize metadata
> > > >
> > > > Note that this use case has been on my mind for more than 2 years:
> > > > https://issues.apache.org/jira/browse/ARROW-1058
> > > >
> > > > I think there are a couple of potentially misleading things at play
> here
> > > >
> > > > 1. The use of the word "protocol". In C, a struct has a well-defined
> > > > binary layout, so a C API is also an ABI. Using C structs to
> > > > communicate data can be considered to be a protocol, but it means
> > > > something different in the context of the "Arrow protocol". I think
> we
> > > > need to call this a "C API"
> > > >
> > > > 2. The documentation for this in Antoine's PR is in the format/
> > > > directory. It would probably be better to have a "C API" section in
> > > > the documentation.
> > > >
> > > > The header file under discussion and the documentation about it is
> > > > best considered as a "library".
> > > >
> > > > It might be useful at some point to create a C99 implementation of
> the
> > > > IPC protocol as well using FlatCC with the goal of having a complete
> > > > implementation of the columnar format in C with minimal binary
> > > > footprint. This is analogous to the NanoPB project which is an
> > > > implementation of Protocol Buffers with small code size
> > > >
> > > > https://github.com/nanopb/nanopb
> > > >
> > > > Let me know if this makes more sense.
> > > >
> > > > I think it's important to communicate clearly about this primarily
> for
> > > > the benefit of the outside world which can confuse easily as we have
> > > > observed over the last few years =)
> > > >
> > > > Wes
> > > >
> > > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> > > wrote:
> > > > >
> > > > > I disagree with this statement:
> > > > >
> > > > > - the IPC format is meant for serialization while the C data
> protocol
> > > is
> > > > > meants for in-memory communication, so different concerns apply
> > > > >
> > > > > If that is how the a particular implementation presents it, that
> is a
> > > > > weaknesses of the implementation, not the format. The primary use
> case
> > > I
> > > > > was focused on when working on the initial format was communication
> > > > within
> > > > > the same process. It seems like this is being used as a basis for
> the
> > > > > introduction of new things when the premise is inconsistent with
> the
> > > > > intention of the creation. The specific reason we used flatbuffers
> in
> > > the
> > > > > project was to collapse the separation of in-process and
> out-of-process
> > > > > communication. It means the same thing it does with the Arrow data
> > > > itself:
> > > > > that a consumer doesn't have to use a particular library to
> interact
> > > with
> > > > > and use the data.
> > > > >
> > > > > It seems like there are two ideas here:
> > > > >
> > > > > 1) How do we make it easier for people to use Arrow?
> > > > > 2) Should we implement a new in memory representation of Arrow
> that is
> > > > > language specific.
> > > > >
> > > > > I'm entirely in support of number one. If for a particular type of
> > > > domain,
> > > > > people want an easier way to interact with Arrow, let's make a new
> > > > library
> > > > > that helps with that. In easy of our current libraries, we do many
> > > things
> > > > > to make it easier to work with Arrow. None of those require a
> change to
> > > > the
> > > > > core format or are formalized as a new in-memory standard. The
> > > in-memory
> > > > > representation of rust or javascript or java objects are
> implementation
> > > > > details.
> > > > >
> > > > > I'm against number two as it creates a fragmentation problem.
> Arrow is
> > > > > about having a single canonical format for memory for both
> metadata and
> > > > > data. Having multiple in-memory formats (especially when some are
> not
> > > > > language independent) is counter to the goals of the project.
> > > >
> > > > I don't think anyone is proposing anything that would cause
> > > fragmentation.
> > > >
> > > > A central question is whether it is useful to define a reusable C ABI
> > > > for the Arrow columnar format, and if there is sufficient interest, a
> > > > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> > > > message) that assembles and disassembles the data structures defined
> > > > in the C ABI.
> > > >
> > > > We could separately create a tiny implementation of the Arrow IPC
> > > > protocol using FlatCC that could be dropped into applications
> > > > requiring only a C compiler and nothing else.
> > > >
> > > >
> > > > >
> > > > > Two other, separate comments:
> > > > > 1) I don't understand the idea that we need to change the way Arrow
> > > > > fundamentally works so that people can avoid using a dependency.
> If the
> > > > > dependency is small, open source and easy to build, people can
> fork it
> > > > and
> > > > > include directly if they want to. Let's not violate project
> principles
> > > > > because DuckDB has a religious perspective on dependencies. If the
> > > > problem
> > > > > is people have to swallow too large of a pill to do basic things
> with
> > > > Arrow
> > > > > in C, let's focus on fixing that (to our definition of ease, not
> > > someone
> > > > > else's). If FlatCC solves some those things, great. If we need to
> > > build a
> > > > > baby integration library that is more C centric, great. Neither of
> > > those
> > > > > things require implementing something at the format level.
> > > > >
> > > > > 2) It seems like we should discuss the data structure problem
> > > separately
> > > > > from the reference management concern.
> > > > >
> > > > >
> > > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi Antoine,
> > > > > >
> > > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <
> antoine@python.org>
> > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > > > A couple things:
> > > > > > > >
> > > > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> > > > better
> > > > > > > > to have the same "shape" as an assembled array. Note that
> the C
> > > > > > > > structs here have very nearly the same "shape" as the data
> > > > structure
> > > > > > > > representing a C++ Array object [1]. The disassembly and
> > > reassembly
> > > > > > > > here is substantially simpler than the IPC protocol. A
> recursive
> > > > > > > > structure in Flatbuffers would make RecordBatch messages much
> > > > larger,
> > > > > > > > so the flattened / disassembled representation we use for
> > > > serialized
> > > > > > > > record batches is the correct one
> > > > > > >
> > > > > > > I'm not sure I agree:
> > > > > > >
> > > > > > > - indeed, it's not a coincidence that the ArrowArray struct
> looks
> > > > quite
> > > > > > > closely like the C++ ArrayData object :-)  We have good
> experience
> > > > with
> > > > > > > that abstraction and it has proven to work quite well
> > > > > > >
> > > > > > > - the IPC format is meant for serialization while the C data
> > > > protocol is
> > > > > > > meants for in-memory communication, so different concerns apply
> > > > > > >
> > > > > > > - the fact that this makes the layout slightly larger doesn't
> seem
> > > > > > > important at all; we're not talking about transferring data
> over
> > > the
> > > > wire
> > > > > > >
> > > > > > > There's also another argument for having a recursive struct: it
> > > > > > > simplifies how the data type is represented, since we can
> encode
> > > each
> > > > > > > child type individually instead of encoding it in the parent's
> > > format
> > > > > > > string (same applies for metadata and individual flags).
> > > > > > >
> > > > > >
> > > > > > I was saying something different here. I was making an argument
> about
> > > > > > why we use the flattened array-of-structs in the IPC protocol.
> One
> > > > > > reason is that it's a more compact representation. That is not
> very
> > > > > > important here because this protocol is only for *in-process*
> (for
> > > > > > languages that have a C FFI facility) rather than *inter-process*
> > > > > > communication.
> > > > > >
> > > > > > I agree also that the type encoding is simple, here, too, since
> we
> > > > > > aren't having to split the schema and record batch between
> different
> > > > > > serialized messages. There is some potential waste with having to
> > > > > > populate the type fields multiple times when communicating a
> sequence
> > > > > > of "chunks" from the same logical dataset.
> > > > > >
> > > > > > > > * The "formal" C protocol having the "assembled" shape means
> that
> > > > many
> > > > > > > > minimal Arrow users won't have to implement any separate data
> > > > > > > > structures. They can just use the C struct directly or a
> slightly
> > > > > > > > wrapped version thereof with some convenience functions.
> > > > > > >
> > > > > > > Yes, but the same applies to the current proposal.
> > > > > > >
> > > > > > > > * I think that requiring building a Flatbuffer for minimal
> use
> > > > cases
> > > > > > > > (e.g. communicating simple record batches with primitive
> types)
> > > > passes
> > > > > > > > on implementation burden to minimal users.
> > > > > > >
> > > > > > > It certainly does.
> > > > > > >
> > > > > > > > I think the mantra of the C protocol should be the following:
> > > > > > > >
> > > > > > > > * Users of the protocol have to write little to no code to
> use
> > > it.
> > > > For
> > > > > > > > example, populating an INT32 array should require only a few
> > > lines
> > > > of
> > > > > > > > code
> > > > > > >
> > > > > > > Agreed.  As a sidenote, the spec should have an example of
> doing
> > > > this in
> > > > > > > raw C.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > Antoine.
> > > > > >
> > > >
> > >
>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

On Wed, Oct 2, 2019 at 11:05 PM Micah Kornfield <em...@gmail.com> wrote:
>
> I've tried to summarize my understanding of the debate so far and give some
> initial thoughts. I think there are two potentially different sets of users
> that we are targeting with a stable C API/ABI ourselves and external
> parties.
>
> 1.  Different language implementations within the Arrow project that want
> to call into each other's code.  We still don't have a great story around
> this in terms of reusable libraries and questions like [1] are a motivating
> examples of making something better in this context.
> 2.  third-parties wishing to support/integrate with Arrow.  Some
> conjectures about these users:
>   - Users in this group are NOT necessarily familiar with existing
> technologies Arrow uses (i.e. flatbuffers)
>   - The stability of the API is the primary concern (consumers don't want
> to change when a new version of the library ships)
>   - An important secondary concern is additional libraries that need to be
> integrated in addition to the API
>
> The main debate points seems to be:
>
> 1.  Vector/Array oriented API vs existing Record Batch.  Will an additional
> column oriented API become too much of a maintenance headache/cause
> fragmentation?
>
>  - In my mind the question here is which set of users we are prioritizing.
> IMO the combination of flatbuffers and translation to/from RecordBatch
> format offers too much friction to make it easy for a third-party
> implementer to use. If we are prioritizing for our own internal use-cases I
> think we should try out a RecordBatch+Flatbuffers based C-API. We already
> have all the necessary building blocks.
>

If a C function passes you a string containing a RecordBatch
Flatbuffers message, what happens next? This message has to be
reassembled into a recursive data structure before you can "do"
anything with it. Are we expecting every third party project to
implement:

A. Data structures appropriate to represent a logical "field" in a
record batch (which have to be recursive to account for nested types'
children)
B. The logic to convert from the flattened Flatbuffers representation
to some implementation of A

I'm arguing that we should provide both to third parties. To build B,
you need A. Some consumers will only use A. This discussion is
essentially about developing an ultraminimalist "drop-in" C
implementation of A.

> 2.  How onerous is the dependency on flat-buffers both from a learning
> curve perspective and as dependency for third-party integrators?
> - Flatbuffers aren't entirely straight-forward and I think if we do move
> forward with an API based on Column/Array we should consider alternatives
> as long as the necessary parsing code can be done in a small amount of code
> (I'm personally against JSON for this, but can see the arguments for it).
>
> 3.  Do all existing library implementations need to support both
> Column/Array a ABI?  How will compliance be checked for the new API/ABI?
>
> - I'm still thinking this through.
>
> [1]
> https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E
>
> On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> > I'd like to hear more opinions from others on this topic. This conversation
> > seems mostly dominated by comments from myself, Wes and Antoine.
> >
> > I think it is reasonable to argue that keeping any ABI (or header/struct
> > pattern) as narrow as possible would allow us to minimize overlap with the
> > existing in-memory specification. In Arrow's case, this could be as simple
> > as a single memory pointer for schema (backed by flatbuffers) and a single
> > memory location for data (that references the record batch header, which in
> > turn provides pointers into the actual arrow data). Extensions would need
> > to be added for reference management as done here but I continue to think
> > we should defer discussion of that until the base data structures are
> > resolved. I see the comments here as arguing for a much broader ABI, in
> > part to support having people build "Arrow" components that interconnect
> > using this new interface. I understand the desire to expand the ABI to be
> > driven by needs to reduce dependencies and ease usability.
> >
> > The representation within the related patch is being presented as a way for
> > applications to share Arrow data but is not easily accessible to all
> > languages. I want to avoid a situation where someone says "I produced an
> > Arrow API" when what they've really done is created a C interface which
> > only a small subset of languages can actually leverage. For example, every
> > language now knows how to parse the existing schema definition as rendered
> > in flatbuf. In order to interact with something that implements this new
> > pattern one would also be required to implement completely new schema
> > consumption code. In the proposal itself it suggests this (for example
> > enhancing the C++ library to consume structures produced this way).
> >
> > As I said, I really want to hear more opinions. Running this past various
> > developers I know, many have echoed my concerns but that really doesn't
> > matter (and who knows how much of that is colored by my presentation of the
> > issue). What do people here think? If someone builds an "Arrow" library
> > that implements this set of structures, how does one use it in Node? In
> > Java? Does it drive creation of a secondary set of interfaces in each of
> > those languages to work with this kind of pattern? (For example, in a JVM
> > view of the world, working with a plain struct in java rather than a set of
> > memory pointers against our existing IPC formats would be quite painful and
> > we'd definitely need to create some glue code for users. I worry the same
> > pattern would occur in many other languages.)
> >
> > To respond directly to some of Wes's most recent comments from the email
> > below. I struggle to map your description of the situation to the rest of
> > the thread and the proposed patch.  For example, you say that a non-goal is
> > "creating a new canonical way to serialize metadata" bute the patch
> > proposes a concrete string based encoding system to describe data types.
> > Aren't those things in conflict?
> >
> > I'll also think more on this and challenge my own perspective. This isn't
> > where my focus is so my comments aren't as developed/thoughtful as I'd
> > like.
> >
> >
> > On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi Jacques,
> > >
> > > I think we've veered off course a bit and maybe we could reframe the
> > > discussion.
> > >
> > > Goals
> > > * A "drop-in" header-only C file that projects can use as a
> > > programming interface either internally only or to expose in-memory
> > > data structures between C functions at call sites. Ideally little to
> > > no disassembly/reassembly should be required on either "side" of the
> > > call site.
> > > * Simplifying adoption of Arrow for C programmers, or languages based
> > > around C FFI
> > >
> > > Non-goals
> > > * Expanding the columnar format or creating an alternative canonical
> > > in-memory representation
> > > * Creating a new canonical way to serialize metadata
> > >
> > > Note that this use case has been on my mind for more than 2 years:
> > > https://issues.apache.org/jira/browse/ARROW-1058
> > >
> > > I think there are a couple of potentially misleading things at play here
> > >
> > > 1. The use of the word "protocol". In C, a struct has a well-defined
> > > binary layout, so a C API is also an ABI. Using C structs to
> > > communicate data can be considered to be a protocol, but it means
> > > something different in the context of the "Arrow protocol". I think we
> > > need to call this a "C API"
> > >
> > > 2. The documentation for this in Antoine's PR is in the format/
> > > directory. It would probably be better to have a "C API" section in
> > > the documentation.
> > >
> > > The header file under discussion and the documentation about it is
> > > best considered as a "library".
> > >
> > > It might be useful at some point to create a C99 implementation of the
> > > IPC protocol as well using FlatCC with the goal of having a complete
> > > implementation of the columnar format in C with minimal binary
> > > footprint. This is analogous to the NanoPB project which is an
> > > implementation of Protocol Buffers with small code size
> > >
> > > https://github.com/nanopb/nanopb
> > >
> > > Let me know if this makes more sense.
> > >
> > > I think it's important to communicate clearly about this primarily for
> > > the benefit of the outside world which can confuse easily as we have
> > > observed over the last few years =)
> > >
> > > Wes
> > >
> > > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> > wrote:
> > > >
> > > > I disagree with this statement:
> > > >
> > > > - the IPC format is meant for serialization while the C data protocol
> > is
> > > > meants for in-memory communication, so different concerns apply
> > > >
> > > > If that is how the a particular implementation presents it, that is a
> > > > weaknesses of the implementation, not the format. The primary use case
> > I
> > > > was focused on when working on the initial format was communication
> > > within
> > > > the same process. It seems like this is being used as a basis for the
> > > > introduction of new things when the premise is inconsistent with the
> > > > intention of the creation. The specific reason we used flatbuffers in
> > the
> > > > project was to collapse the separation of in-process and out-of-process
> > > > communication. It means the same thing it does with the Arrow data
> > > itself:
> > > > that a consumer doesn't have to use a particular library to interact
> > with
> > > > and use the data.
> > > >
> > > > It seems like there are two ideas here:
> > > >
> > > > 1) How do we make it easier for people to use Arrow?
> > > > 2) Should we implement a new in memory representation of Arrow that is
> > > > language specific.
> > > >
> > > > I'm entirely in support of number one. If for a particular type of
> > > domain,
> > > > people want an easier way to interact with Arrow, let's make a new
> > > library
> > > > that helps with that. In easy of our current libraries, we do many
> > things
> > > > to make it easier to work with Arrow. None of those require a change to
> > > the
> > > > core format or are formalized as a new in-memory standard. The
> > in-memory
> > > > representation of rust or javascript or java objects are implementation
> > > > details.
> > > >
> > > > I'm against number two as it creates a fragmentation problem. Arrow is
> > > > about having a single canonical format for memory for both metadata and
> > > > data. Having multiple in-memory formats (especially when some are not
> > > > language independent) is counter to the goals of the project.
> > >
> > > I don't think anyone is proposing anything that would cause
> > fragmentation.
> > >
> > > A central question is whether it is useful to define a reusable C ABI
> > > for the Arrow columnar format, and if there is sufficient interest, a
> > > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> > > message) that assembles and disassembles the data structures defined
> > > in the C ABI.
> > >
> > > We could separately create a tiny implementation of the Arrow IPC
> > > protocol using FlatCC that could be dropped into applications
> > > requiring only a C compiler and nothing else.
> > >
> > >
> > > >
> > > > Two other, separate comments:
> > > > 1) I don't understand the idea that we need to change the way Arrow
> > > > fundamentally works so that people can avoid using a dependency. If the
> > > > dependency is small, open source and easy to build, people can fork it
> > > and
> > > > include directly if they want to. Let's not violate project principles
> > > > because DuckDB has a religious perspective on dependencies. If the
> > > problem
> > > > is people have to swallow too large of a pill to do basic things with
> > > Arrow
> > > > in C, let's focus on fixing that (to our definition of ease, not
> > someone
> > > > else's). If FlatCC solves some those things, great. If we need to
> > build a
> > > > baby integration library that is more C centric, great. Neither of
> > those
> > > > things require implementing something at the format level.
> > > >
> > > > 2) It seems like we should discuss the data structure problem
> > separately
> > > > from the reference management concern.
> > > >
> > > >
> > > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > hi Antoine,
> > > > >
> > > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org>
> > > wrote:
> > > > > >
> > > > > >
> > > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > > A couple things:
> > > > > > >
> > > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> > > better
> > > > > > > to have the same "shape" as an assembled array. Note that the C
> > > > > > > structs here have very nearly the same "shape" as the data
> > > structure
> > > > > > > representing a C++ Array object [1]. The disassembly and
> > reassembly
> > > > > > > here is substantially simpler than the IPC protocol. A recursive
> > > > > > > structure in Flatbuffers would make RecordBatch messages much
> > > larger,
> > > > > > > so the flattened / disassembled representation we use for
> > > serialized
> > > > > > > record batches is the correct one
> > > > > >
> > > > > > I'm not sure I agree:
> > > > > >
> > > > > > - indeed, it's not a coincidence that the ArrowArray struct looks
> > > quite
> > > > > > closely like the C++ ArrayData object :-)  We have good experience
> > > with
> > > > > > that abstraction and it has proven to work quite well
> > > > > >
> > > > > > - the IPC format is meant for serialization while the C data
> > > protocol is
> > > > > > meants for in-memory communication, so different concerns apply
> > > > > >
> > > > > > - the fact that this makes the layout slightly larger doesn't seem
> > > > > > important at all; we're not talking about transferring data over
> > the
> > > wire
> > > > > >
> > > > > > There's also another argument for having a recursive struct: it
> > > > > > simplifies how the data type is represented, since we can encode
> > each
> > > > > > child type individually instead of encoding it in the parent's
> > format
> > > > > > string (same applies for metadata and individual flags).
> > > > > >
> > > > >
> > > > > I was saying something different here. I was making an argument about
> > > > > why we use the flattened array-of-structs in the IPC protocol. One
> > > > > reason is that it's a more compact representation. That is not very
> > > > > important here because this protocol is only for *in-process* (for
> > > > > languages that have a C FFI facility) rather than *inter-process*
> > > > > communication.
> > > > >
> > > > > I agree also that the type encoding is simple, here, too, since we
> > > > > aren't having to split the schema and record batch between different
> > > > > serialized messages. There is some potential waste with having to
> > > > > populate the type fields multiple times when communicating a sequence
> > > > > of "chunks" from the same logical dataset.
> > > > >
> > > > > > > * The "formal" C protocol having the "assembled" shape means that
> > > many
> > > > > > > minimal Arrow users won't have to implement any separate data
> > > > > > > structures. They can just use the C struct directly or a slightly
> > > > > > > wrapped version thereof with some convenience functions.
> > > > > >
> > > > > > Yes, but the same applies to the current proposal.
> > > > > >
> > > > > > > * I think that requiring building a Flatbuffer for minimal use
> > > cases
> > > > > > > (e.g. communicating simple record batches with primitive types)
> > > passes
> > > > > > > on implementation burden to minimal users.
> > > > > >
> > > > > > It certainly does.
> > > > > >
> > > > > > > I think the mantra of the C protocol should be the following:
> > > > > > >
> > > > > > > * Users of the protocol have to write little to no code to use
> > it.
> > > For
> > > > > > > example, populating an INT32 array should require only a few
> > lines
> > > of
> > > > > > > code
> > > > > >
> > > > > > Agreed.  As a sidenote, the spec should have an example of doing
> > > this in
> > > > > > raw C.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Antoine.
> > > > >
> > >
> >

Re: [DISCUSS] C-level in-process array protocol

[Micah Kornfield <em...@gmail.com>]

I've tried to summarize my understanding of the debate so far and give some
initial thoughts. I think there are two potentially different sets of users
that we are targeting with a stable C API/ABI ourselves and external
parties.

1.  Different language implementations within the Arrow project that want
to call into each other's code.  We still don't have a great story around
this in terms of reusable libraries and questions like [1] are a motivating
examples of making something better in this context.
2.  third-parties wishing to support/integrate with Arrow.  Some
conjectures about these users:
  - Users in this group are NOT necessarily familiar with existing
technologies Arrow uses (i.e. flatbuffers)
  - The stability of the API is the primary concern (consumers don't want
to change when a new version of the library ships)
  - An important secondary concern is additional libraries that need to be
integrated in addition to the API

The main debate points seems to be:

1.  Vector/Array oriented API vs existing Record Batch.  Will an additional
column oriented API become too much of a maintenance headache/cause
fragmentation?

 - In my mind the question here is which set of users we are prioritizing.
IMO the combination of flatbuffers and translation to/from RecordBatch
format offers too much friction to make it easy for a third-party
implementer to use. If we are prioritizing for our own internal use-cases I
think we should try out a RecordBatch+Flatbuffers based C-API. We already
have all the necessary building blocks.

2.  How onerous is the dependency on flat-buffers both from a learning
curve perspective and as dependency for third-party integrators?
- Flatbuffers aren't entirely straight-forward and I think if we do move
forward with an API based on Column/Array we should consider alternatives
as long as the necessary parsing code can be done in a small amount of code
(I'm personally against JSON for this, but can see the arguments for it).

3.  Do all existing library implementations need to support both
Column/Array a ABI?  How will compliance be checked for the new API/ABI?

- I'm still thinking this through.

[1]
https://lists.apache.org/thread.html/18244b294d0b9bd568b5cfd1b1ac2b6a25088383a08202cc7a8a3563@%3Cuser.arrow.apache.org%3E

On Wed, Oct 2, 2019 at 6:46 PM Jacques Nadeau <ja...@apache.org> wrote:

> I'd like to hear more opinions from others on this topic. This conversation
> seems mostly dominated by comments from myself, Wes and Antoine.
>
> I think it is reasonable to argue that keeping any ABI (or header/struct
> pattern) as narrow as possible would allow us to minimize overlap with the
> existing in-memory specification. In Arrow's case, this could be as simple
> as a single memory pointer for schema (backed by flatbuffers) and a single
> memory location for data (that references the record batch header, which in
> turn provides pointers into the actual arrow data). Extensions would need
> to be added for reference management as done here but I continue to think
> we should defer discussion of that until the base data structures are
> resolved. I see the comments here as arguing for a much broader ABI, in
> part to support having people build "Arrow" components that interconnect
> using this new interface. I understand the desire to expand the ABI to be
> driven by needs to reduce dependencies and ease usability.
>
> The representation within the related patch is being presented as a way for
> applications to share Arrow data but is not easily accessible to all
> languages. I want to avoid a situation where someone says "I produced an
> Arrow API" when what they've really done is created a C interface which
> only a small subset of languages can actually leverage. For example, every
> language now knows how to parse the existing schema definition as rendered
> in flatbuf. In order to interact with something that implements this new
> pattern one would also be required to implement completely new schema
> consumption code. In the proposal itself it suggests this (for example
> enhancing the C++ library to consume structures produced this way).
>
> As I said, I really want to hear more opinions. Running this past various
> developers I know, many have echoed my concerns but that really doesn't
> matter (and who knows how much of that is colored by my presentation of the
> issue). What do people here think? If someone builds an "Arrow" library
> that implements this set of structures, how does one use it in Node? In
> Java? Does it drive creation of a secondary set of interfaces in each of
> those languages to work with this kind of pattern? (For example, in a JVM
> view of the world, working with a plain struct in java rather than a set of
> memory pointers against our existing IPC formats would be quite painful and
> we'd definitely need to create some glue code for users. I worry the same
> pattern would occur in many other languages.)
>
> To respond directly to some of Wes's most recent comments from the email
> below. I struggle to map your description of the situation to the rest of
> the thread and the proposed patch.  For example, you say that a non-goal is
> "creating a new canonical way to serialize metadata" bute the patch
> proposes a concrete string based encoding system to describe data types.
> Aren't those things in conflict?
>
> I'll also think more on this and challenge my own perspective. This isn't
> where my focus is so my comments aren't as developed/thoughtful as I'd
> like.
>
>
> On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Jacques,
> >
> > I think we've veered off course a bit and maybe we could reframe the
> > discussion.
> >
> > Goals
> > * A "drop-in" header-only C file that projects can use as a
> > programming interface either internally only or to expose in-memory
> > data structures between C functions at call sites. Ideally little to
> > no disassembly/reassembly should be required on either "side" of the
> > call site.
> > * Simplifying adoption of Arrow for C programmers, or languages based
> > around C FFI
> >
> > Non-goals
> > * Expanding the columnar format or creating an alternative canonical
> > in-memory representation
> > * Creating a new canonical way to serialize metadata
> >
> > Note that this use case has been on my mind for more than 2 years:
> > https://issues.apache.org/jira/browse/ARROW-1058
> >
> > I think there are a couple of potentially misleading things at play here
> >
> > 1. The use of the word "protocol". In C, a struct has a well-defined
> > binary layout, so a C API is also an ABI. Using C structs to
> > communicate data can be considered to be a protocol, but it means
> > something different in the context of the "Arrow protocol". I think we
> > need to call this a "C API"
> >
> > 2. The documentation for this in Antoine's PR is in the format/
> > directory. It would probably be better to have a "C API" section in
> > the documentation.
> >
> > The header file under discussion and the documentation about it is
> > best considered as a "library".
> >
> > It might be useful at some point to create a C99 implementation of the
> > IPC protocol as well using FlatCC with the goal of having a complete
> > implementation of the columnar format in C with minimal binary
> > footprint. This is analogous to the NanoPB project which is an
> > implementation of Protocol Buffers with small code size
> >
> > https://github.com/nanopb/nanopb
> >
> > Let me know if this makes more sense.
> >
> > I think it's important to communicate clearly about this primarily for
> > the benefit of the outside world which can confuse easily as we have
> > observed over the last few years =)
> >
> > Wes
> >
> > On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org>
> wrote:
> > >
> > > I disagree with this statement:
> > >
> > > - the IPC format is meant for serialization while the C data protocol
> is
> > > meants for in-memory communication, so different concerns apply
> > >
> > > If that is how the a particular implementation presents it, that is a
> > > weaknesses of the implementation, not the format. The primary use case
> I
> > > was focused on when working on the initial format was communication
> > within
> > > the same process. It seems like this is being used as a basis for the
> > > introduction of new things when the premise is inconsistent with the
> > > intention of the creation. The specific reason we used flatbuffers in
> the
> > > project was to collapse the separation of in-process and out-of-process
> > > communication. It means the same thing it does with the Arrow data
> > itself:
> > > that a consumer doesn't have to use a particular library to interact
> with
> > > and use the data.
> > >
> > > It seems like there are two ideas here:
> > >
> > > 1) How do we make it easier for people to use Arrow?
> > > 2) Should we implement a new in memory representation of Arrow that is
> > > language specific.
> > >
> > > I'm entirely in support of number one. If for a particular type of
> > domain,
> > > people want an easier way to interact with Arrow, let's make a new
> > library
> > > that helps with that. In easy of our current libraries, we do many
> things
> > > to make it easier to work with Arrow. None of those require a change to
> > the
> > > core format or are formalized as a new in-memory standard. The
> in-memory
> > > representation of rust or javascript or java objects are implementation
> > > details.
> > >
> > > I'm against number two as it creates a fragmentation problem. Arrow is
> > > about having a single canonical format for memory for both metadata and
> > > data. Having multiple in-memory formats (especially when some are not
> > > language independent) is counter to the goals of the project.
> >
> > I don't think anyone is proposing anything that would cause
> fragmentation.
> >
> > A central question is whether it is useful to define a reusable C ABI
> > for the Arrow columnar format, and if there is sufficient interest, a
> > tiny C implementation of the IPC protocol (which uses the Flatbuffers
> > message) that assembles and disassembles the data structures defined
> > in the C ABI.
> >
> > We could separately create a tiny implementation of the Arrow IPC
> > protocol using FlatCC that could be dropped into applications
> > requiring only a C compiler and nothing else.
> >
> >
> > >
> > > Two other, separate comments:
> > > 1) I don't understand the idea that we need to change the way Arrow
> > > fundamentally works so that people can avoid using a dependency. If the
> > > dependency is small, open source and easy to build, people can fork it
> > and
> > > include directly if they want to. Let's not violate project principles
> > > because DuckDB has a religious perspective on dependencies. If the
> > problem
> > > is people have to swallow too large of a pill to do basic things with
> > Arrow
> > > in C, let's focus on fixing that (to our definition of ease, not
> someone
> > > else's). If FlatCC solves some those things, great. If we need to
> build a
> > > baby integration library that is more C centric, great. Neither of
> those
> > > things require implementing something at the format level.
> > >
> > > 2) It seems like we should discuss the data structure problem
> separately
> > > from the reference management concern.
> > >
> > >
> > > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > hi Antoine,
> > > >
> > > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > > > >
> > > > >
> > > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > > A couple things:
> > > > > >
> > > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> > better
> > > > > > to have the same "shape" as an assembled array. Note that the C
> > > > > > structs here have very nearly the same "shape" as the data
> > structure
> > > > > > representing a C++ Array object [1]. The disassembly and
> reassembly
> > > > > > here is substantially simpler than the IPC protocol. A recursive
> > > > > > structure in Flatbuffers would make RecordBatch messages much
> > larger,
> > > > > > so the flattened / disassembled representation we use for
> > serialized
> > > > > > record batches is the correct one
> > > > >
> > > > > I'm not sure I agree:
> > > > >
> > > > > - indeed, it's not a coincidence that the ArrowArray struct looks
> > quite
> > > > > closely like the C++ ArrayData object :-)  We have good experience
> > with
> > > > > that abstraction and it has proven to work quite well
> > > > >
> > > > > - the IPC format is meant for serialization while the C data
> > protocol is
> > > > > meants for in-memory communication, so different concerns apply
> > > > >
> > > > > - the fact that this makes the layout slightly larger doesn't seem
> > > > > important at all; we're not talking about transferring data over
> the
> > wire
> > > > >
> > > > > There's also another argument for having a recursive struct: it
> > > > > simplifies how the data type is represented, since we can encode
> each
> > > > > child type individually instead of encoding it in the parent's
> format
> > > > > string (same applies for metadata and individual flags).
> > > > >
> > > >
> > > > I was saying something different here. I was making an argument about
> > > > why we use the flattened array-of-structs in the IPC protocol. One
> > > > reason is that it's a more compact representation. That is not very
> > > > important here because this protocol is only for *in-process* (for
> > > > languages that have a C FFI facility) rather than *inter-process*
> > > > communication.
> > > >
> > > > I agree also that the type encoding is simple, here, too, since we
> > > > aren't having to split the schema and record batch between different
> > > > serialized messages. There is some potential waste with having to
> > > > populate the type fields multiple times when communicating a sequence
> > > > of "chunks" from the same logical dataset.
> > > >
> > > > > > * The "formal" C protocol having the "assembled" shape means that
> > many
> > > > > > minimal Arrow users won't have to implement any separate data
> > > > > > structures. They can just use the C struct directly or a slightly
> > > > > > wrapped version thereof with some convenience functions.
> > > > >
> > > > > Yes, but the same applies to the current proposal.
> > > > >
> > > > > > * I think that requiring building a Flatbuffer for minimal use
> > cases
> > > > > > (e.g. communicating simple record batches with primitive types)
> > passes
> > > > > > on implementation burden to minimal users.
> > > > >
> > > > > It certainly does.
> > > > >
> > > > > > I think the mantra of the C protocol should be the following:
> > > > > >
> > > > > > * Users of the protocol have to write little to no code to use
> it.
> > For
> > > > > > example, populating an INT32 array should require only a few
> lines
> > of
> > > > > > code
> > > > >
> > > > > Agreed.  As a sidenote, the spec should have an example of doing
> > this in
> > > > > raw C.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > >
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

I'd like to hear more opinions from others on this topic. This conversation
seems mostly dominated by comments from myself, Wes and Antoine.

I think it is reasonable to argue that keeping any ABI (or header/struct
pattern) as narrow as possible would allow us to minimize overlap with the
existing in-memory specification. In Arrow's case, this could be as simple
as a single memory pointer for schema (backed by flatbuffers) and a single
memory location for data (that references the record batch header, which in
turn provides pointers into the actual arrow data). Extensions would need
to be added for reference management as done here but I continue to think
we should defer discussion of that until the base data structures are
resolved. I see the comments here as arguing for a much broader ABI, in
part to support having people build "Arrow" components that interconnect
using this new interface. I understand the desire to expand the ABI to be
driven by needs to reduce dependencies and ease usability.

The representation within the related patch is being presented as a way for
applications to share Arrow data but is not easily accessible to all
languages. I want to avoid a situation where someone says "I produced an
Arrow API" when what they've really done is created a C interface which
only a small subset of languages can actually leverage. For example, every
language now knows how to parse the existing schema definition as rendered
in flatbuf. In order to interact with something that implements this new
pattern one would also be required to implement completely new schema
consumption code. In the proposal itself it suggests this (for example
enhancing the C++ library to consume structures produced this way).

As I said, I really want to hear more opinions. Running this past various
developers I know, many have echoed my concerns but that really doesn't
matter (and who knows how much of that is colored by my presentation of the
issue). What do people here think? If someone builds an "Arrow" library
that implements this set of structures, how does one use it in Node? In
Java? Does it drive creation of a secondary set of interfaces in each of
those languages to work with this kind of pattern? (For example, in a JVM
view of the world, working with a plain struct in java rather than a set of
memory pointers against our existing IPC formats would be quite painful and
we'd definitely need to create some glue code for users. I worry the same
pattern would occur in many other languages.)

To respond directly to some of Wes's most recent comments from the email
below. I struggle to map your description of the situation to the rest of
the thread and the proposed patch.  For example, you say that a non-goal is
"creating a new canonical way to serialize metadata" bute the patch
proposes a concrete string based encoding system to describe data types.
Aren't those things in conflict?

I'll also think more on this and challenge my own perspective. This isn't
where my focus is so my comments aren't as developed/thoughtful as I'd like.


On Tue, Oct 1, 2019 at 7:33 PM Wes McKinney <we...@gmail.com> wrote:

> hi Jacques,
>
> I think we've veered off course a bit and maybe we could reframe the
> discussion.
>
> Goals
> * A "drop-in" header-only C file that projects can use as a
> programming interface either internally only or to expose in-memory
> data structures between C functions at call sites. Ideally little to
> no disassembly/reassembly should be required on either "side" of the
> call site.
> * Simplifying adoption of Arrow for C programmers, or languages based
> around C FFI
>
> Non-goals
> * Expanding the columnar format or creating an alternative canonical
> in-memory representation
> * Creating a new canonical way to serialize metadata
>
> Note that this use case has been on my mind for more than 2 years:
> https://issues.apache.org/jira/browse/ARROW-1058
>
> I think there are a couple of potentially misleading things at play here
>
> 1. The use of the word "protocol". In C, a struct has a well-defined
> binary layout, so a C API is also an ABI. Using C structs to
> communicate data can be considered to be a protocol, but it means
> something different in the context of the "Arrow protocol". I think we
> need to call this a "C API"
>
> 2. The documentation for this in Antoine's PR is in the format/
> directory. It would probably be better to have a "C API" section in
> the documentation.
>
> The header file under discussion and the documentation about it is
> best considered as a "library".
>
> It might be useful at some point to create a C99 implementation of the
> IPC protocol as well using FlatCC with the goal of having a complete
> implementation of the columnar format in C with minimal binary
> footprint. This is analogous to the NanoPB project which is an
> implementation of Protocol Buffers with small code size
>
> https://github.com/nanopb/nanopb
>
> Let me know if this makes more sense.
>
> I think it's important to communicate clearly about this primarily for
> the benefit of the outside world which can confuse easily as we have
> observed over the last few years =)
>
> Wes
>
> On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org> wrote:
> >
> > I disagree with this statement:
> >
> > - the IPC format is meant for serialization while the C data protocol is
> > meants for in-memory communication, so different concerns apply
> >
> > If that is how the a particular implementation presents it, that is a
> > weaknesses of the implementation, not the format. The primary use case I
> > was focused on when working on the initial format was communication
> within
> > the same process. It seems like this is being used as a basis for the
> > introduction of new things when the premise is inconsistent with the
> > intention of the creation. The specific reason we used flatbuffers in the
> > project was to collapse the separation of in-process and out-of-process
> > communication. It means the same thing it does with the Arrow data
> itself:
> > that a consumer doesn't have to use a particular library to interact with
> > and use the data.
> >
> > It seems like there are two ideas here:
> >
> > 1) How do we make it easier for people to use Arrow?
> > 2) Should we implement a new in memory representation of Arrow that is
> > language specific.
> >
> > I'm entirely in support of number one. If for a particular type of
> domain,
> > people want an easier way to interact with Arrow, let's make a new
> library
> > that helps with that. In easy of our current libraries, we do many things
> > to make it easier to work with Arrow. None of those require a change to
> the
> > core format or are formalized as a new in-memory standard. The in-memory
> > representation of rust or javascript or java objects are implementation
> > details.
> >
> > I'm against number two as it creates a fragmentation problem. Arrow is
> > about having a single canonical format for memory for both metadata and
> > data. Having multiple in-memory formats (especially when some are not
> > language independent) is counter to the goals of the project.
>
> I don't think anyone is proposing anything that would cause fragmentation.
>
> A central question is whether it is useful to define a reusable C ABI
> for the Arrow columnar format, and if there is sufficient interest, a
> tiny C implementation of the IPC protocol (which uses the Flatbuffers
> message) that assembles and disassembles the data structures defined
> in the C ABI.
>
> We could separately create a tiny implementation of the Arrow IPC
> protocol using FlatCC that could be dropped into applications
> requiring only a C compiler and nothing else.
>
>
> >
> > Two other, separate comments:
> > 1) I don't understand the idea that we need to change the way Arrow
> > fundamentally works so that people can avoid using a dependency. If the
> > dependency is small, open source and easy to build, people can fork it
> and
> > include directly if they want to. Let's not violate project principles
> > because DuckDB has a religious perspective on dependencies. If the
> problem
> > is people have to swallow too large of a pill to do basic things with
> Arrow
> > in C, let's focus on fixing that (to our definition of ease, not someone
> > else's). If FlatCC solves some those things, great. If we need to build a
> > baby integration library that is more C centric, great. Neither of those
> > things require implementing something at the format level.
> >
> > 2) It seems like we should discuss the data structure problem separately
> > from the reference management concern.
> >
> >
> > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi Antoine,
> > >
> > > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org>
> wrote:
> > > >
> > > >
> > > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > > A couple things:
> > > > >
> > > > > * I think a C protocol / FFI for Arrow array/vectors would be
> better
> > > > > to have the same "shape" as an assembled array. Note that the C
> > > > > structs here have very nearly the same "shape" as the data
> structure
> > > > > representing a C++ Array object [1]. The disassembly and reassembly
> > > > > here is substantially simpler than the IPC protocol. A recursive
> > > > > structure in Flatbuffers would make RecordBatch messages much
> larger,
> > > > > so the flattened / disassembled representation we use for
> serialized
> > > > > record batches is the correct one
> > > >
> > > > I'm not sure I agree:
> > > >
> > > > - indeed, it's not a coincidence that the ArrowArray struct looks
> quite
> > > > closely like the C++ ArrayData object :-)  We have good experience
> with
> > > > that abstraction and it has proven to work quite well
> > > >
> > > > - the IPC format is meant for serialization while the C data
> protocol is
> > > > meants for in-memory communication, so different concerns apply
> > > >
> > > > - the fact that this makes the layout slightly larger doesn't seem
> > > > important at all; we're not talking about transferring data over the
> wire
> > > >
> > > > There's also another argument for having a recursive struct: it
> > > > simplifies how the data type is represented, since we can encode each
> > > > child type individually instead of encoding it in the parent's format
> > > > string (same applies for metadata and individual flags).
> > > >
> > >
> > > I was saying something different here. I was making an argument about
> > > why we use the flattened array-of-structs in the IPC protocol. One
> > > reason is that it's a more compact representation. That is not very
> > > important here because this protocol is only for *in-process* (for
> > > languages that have a C FFI facility) rather than *inter-process*
> > > communication.
> > >
> > > I agree also that the type encoding is simple, here, too, since we
> > > aren't having to split the schema and record batch between different
> > > serialized messages. There is some potential waste with having to
> > > populate the type fields multiple times when communicating a sequence
> > > of "chunks" from the same logical dataset.
> > >
> > > > > * The "formal" C protocol having the "assembled" shape means that
> many
> > > > > minimal Arrow users won't have to implement any separate data
> > > > > structures. They can just use the C struct directly or a slightly
> > > > > wrapped version thereof with some convenience functions.
> > > >
> > > > Yes, but the same applies to the current proposal.
> > > >
> > > > > * I think that requiring building a Flatbuffer for minimal use
> cases
> > > > > (e.g. communicating simple record batches with primitive types)
> passes
> > > > > on implementation burden to minimal users.
> > > >
> > > > It certainly does.
> > > >
> > > > > I think the mantra of the C protocol should be the following:
> > > > >
> > > > > * Users of the protocol have to write little to no code to use it.
> For
> > > > > example, populating an INT32 array should require only a few lines
> of
> > > > > code
> > > >
> > > > Agreed.  As a sidenote, the spec should have an example of doing
> this in
> > > > raw C.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > >
>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

I had an e-mail editing snafu so you can ignore the bottom "inline"
portion since it's just a restatement of what is written more clearly
above

On Tue, Oct 1, 2019 at 9:32 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi Jacques,
>
> I think we've veered off course a bit and maybe we could reframe the discussion.
>
> Goals
> * A "drop-in" header-only C file that projects can use as a
> programming interface either internally only or to expose in-memory
> data structures between C functions at call sites. Ideally little to
> no disassembly/reassembly should be required on either "side" of the
> call site.
> * Simplifying adoption of Arrow for C programmers, or languages based
> around C FFI
>
> Non-goals
> * Expanding the columnar format or creating an alternative canonical
> in-memory representation
> * Creating a new canonical way to serialize metadata
>
> Note that this use case has been on my mind for more than 2 years:
> https://issues.apache.org/jira/browse/ARROW-1058
>
> I think there are a couple of potentially misleading things at play here
>
> 1. The use of the word "protocol". In C, a struct has a well-defined
> binary layout, so a C API is also an ABI. Using C structs to
> communicate data can be considered to be a protocol, but it means
> something different in the context of the "Arrow protocol". I think we
> need to call this a "C API"
>
> 2. The documentation for this in Antoine's PR is in the format/
> directory. It would probably be better to have a "C API" section in
> the documentation.
>
> The header file under discussion and the documentation about it is
> best considered as a "library".
>
> It might be useful at some point to create a C99 implementation of the
> IPC protocol as well using FlatCC with the goal of having a complete
> implementation of the columnar format in C with minimal binary
> footprint. This is analogous to the NanoPB project which is an
> implementation of Protocol Buffers with small code size
>
> https://github.com/nanopb/nanopb
>
> Let me know if this makes more sense.
>
> I think it's important to communicate clearly about this primarily for
> the benefit of the outside world which can confuse easily as we have
> observed over the last few years =)
>
> Wes
>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

hi Jacques,

I think we've veered off course a bit and maybe we could reframe the discussion.

Goals
* A "drop-in" header-only C file that projects can use as a
programming interface either internally only or to expose in-memory
data structures between C functions at call sites. Ideally little to
no disassembly/reassembly should be required on either "side" of the
call site.
* Simplifying adoption of Arrow for C programmers, or languages based
around C FFI

Non-goals
* Expanding the columnar format or creating an alternative canonical
in-memory representation
* Creating a new canonical way to serialize metadata

Note that this use case has been on my mind for more than 2 years:
https://issues.apache.org/jira/browse/ARROW-1058

I think there are a couple of potentially misleading things at play here

1. The use of the word "protocol". In C, a struct has a well-defined
binary layout, so a C API is also an ABI. Using C structs to
communicate data can be considered to be a protocol, but it means
something different in the context of the "Arrow protocol". I think we
need to call this a "C API"

2. The documentation for this in Antoine's PR is in the format/
directory. It would probably be better to have a "C API" section in
the documentation.

The header file under discussion and the documentation about it is
best considered as a "library".

It might be useful at some point to create a C99 implementation of the
IPC protocol as well using FlatCC with the goal of having a complete
implementation of the columnar format in C with minimal binary
footprint. This is analogous to the NanoPB project which is an
implementation of Protocol Buffers with small code size

https://github.com/nanopb/nanopb

Let me know if this makes more sense.

I think it's important to communicate clearly about this primarily for
the benefit of the outside world which can confuse easily as we have
observed over the last few years =)

Wes

On Tue, Oct 1, 2019 at 2:55 PM Jacques Nadeau <ja...@apache.org> wrote:
>
> I disagree with this statement:
>
> - the IPC format is meant for serialization while the C data protocol is
> meants for in-memory communication, so different concerns apply
>
> If that is how the a particular implementation presents it, that is a
> weaknesses of the implementation, not the format. The primary use case I
> was focused on when working on the initial format was communication within
> the same process. It seems like this is being used as a basis for the
> introduction of new things when the premise is inconsistent with the
> intention of the creation. The specific reason we used flatbuffers in the
> project was to collapse the separation of in-process and out-of-process
> communication. It means the same thing it does with the Arrow data itself:
> that a consumer doesn't have to use a particular library to interact with
> and use the data.
>
> It seems like there are two ideas here:
>
> 1) How do we make it easier for people to use Arrow?
> 2) Should we implement a new in memory representation of Arrow that is
> language specific.
>
> I'm entirely in support of number one. If for a particular type of domain,
> people want an easier way to interact with Arrow, let's make a new library
> that helps with that. In easy of our current libraries, we do many things
> to make it easier to work with Arrow. None of those require a change to the
> core format or are formalized as a new in-memory standard. The in-memory
> representation of rust or javascript or java objects are implementation
> details.
>
> I'm against number two as it creates a fragmentation problem. Arrow is
> about having a single canonical format for memory for both metadata and
> data. Having multiple in-memory formats (especially when some are not
> language independent) is counter to the goals of the project.

I don't think anyone is proposing anything that would cause fragmentation.

A central question is whether it is useful to define a reusable C ABI
for the Arrow columnar format, and if there is sufficient interest, a
tiny C implementation of the IPC protocol (which uses the Flatbuffers
message) that assembles and disassembles the data structures defined
in the C ABI.

We could separately create a tiny implementation of the Arrow IPC
protocol using FlatCC that could be dropped into applications
requiring only a C compiler and nothing else.


>
> Two other, separate comments:
> 1) I don't understand the idea that we need to change the way Arrow
> fundamentally works so that people can avoid using a dependency. If the
> dependency is small, open source and easy to build, people can fork it and
> include directly if they want to. Let's not violate project principles
> because DuckDB has a religious perspective on dependencies. If the problem
> is people have to swallow too large of a pill to do basic things with Arrow
> in C, let's focus on fixing that (to our definition of ease, not someone
> else's). If FlatCC solves some those things, great. If we need to build a
> baby integration library that is more C centric, great. Neither of those
> things require implementing something at the format level.
>
> 2) It seems like we should discuss the data structure problem separately
> from the reference management concern.
>
>
> On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Antoine,
> >
> > On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org> wrote:
> > >
> > >
> > > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > > A couple things:
> > > >
> > > > * I think a C protocol / FFI for Arrow array/vectors would be better
> > > > to have the same "shape" as an assembled array. Note that the C
> > > > structs here have very nearly the same "shape" as the data structure
> > > > representing a C++ Array object [1]. The disassembly and reassembly
> > > > here is substantially simpler than the IPC protocol. A recursive
> > > > structure in Flatbuffers would make RecordBatch messages much larger,
> > > > so the flattened / disassembled representation we use for serialized
> > > > record batches is the correct one
> > >
> > > I'm not sure I agree:
> > >
> > > - indeed, it's not a coincidence that the ArrowArray struct looks quite
> > > closely like the C++ ArrayData object :-)  We have good experience with
> > > that abstraction and it has proven to work quite well
> > >
> > > - the IPC format is meant for serialization while the C data protocol is
> > > meants for in-memory communication, so different concerns apply
> > >
> > > - the fact that this makes the layout slightly larger doesn't seem
> > > important at all; we're not talking about transferring data over the wire
> > >
> > > There's also another argument for having a recursive struct: it
> > > simplifies how the data type is represented, since we can encode each
> > > child type individually instead of encoding it in the parent's format
> > > string (same applies for metadata and individual flags).
> > >
> >
> > I was saying something different here. I was making an argument about
> > why we use the flattened array-of-structs in the IPC protocol. One
> > reason is that it's a more compact representation. That is not very
> > important here because this protocol is only for *in-process* (for
> > languages that have a C FFI facility) rather than *inter-process*
> > communication.
> >
> > I agree also that the type encoding is simple, here, too, since we
> > aren't having to split the schema and record batch between different
> > serialized messages. There is some potential waste with having to
> > populate the type fields multiple times when communicating a sequence
> > of "chunks" from the same logical dataset.
> >
> > > > * The "formal" C protocol having the "assembled" shape means that many
> > > > minimal Arrow users won't have to implement any separate data
> > > > structures. They can just use the C struct directly or a slightly
> > > > wrapped version thereof with some convenience functions.
> > >
> > > Yes, but the same applies to the current proposal.
> > >
> > > > * I think that requiring building a Flatbuffer for minimal use cases
> > > > (e.g. communicating simple record batches with primitive types) passes
> > > > on implementation burden to minimal users.
> > >
> > > It certainly does.
> > >
> > > > I think the mantra of the C protocol should be the following:
> > > >
> > > > * Users of the protocol have to write little to no code to use it. For
> > > > example, populating an INT32 array should require only a few lines of
> > > > code
> > >
> > > Agreed.  As a sidenote, the spec should have an example of doing this in
> > > raw C.
> > >
> > > Regards
> > >
> > > Antoine.
> >

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

On Tue, Oct 1, 2019 at 3:22 PM Jed Brown <je...@jedbrown.org> wrote:
>
> I'd just like to chime in with the use case of in-situ data analysis for
> simulations.  This domain tends to be cautious with dependencies and
> there is a lot of C and Fortran, but the in-situ analysis tools will
> preferably reside in separate processes while sharing memory via shared
> memory (/dev/shm or MPI_Win_allocate_shared).  An in-memory protocol
> that holds raw pointers would be problematic because they are typically
> in different virtual address spaces when shared between processes.  I
> think this is a potential application for a C interface with lean
> dependencies, but it wouldn't be useful if it can't be shared
> out-of-process.
>

hi Jed -- I will respond to Jacques's e-mail when I have some time to
compose, but we're looking at a pretty different use case which is C
libraries (or libraries with C FFI) exposing in-memory data structures
to each other in the same process using the same virtual address
space.

- Wes

> Jacques Nadeau <ja...@apache.org> writes:
>
> > I disagree with this statement:
> >
> > - the IPC format is meant for serialization while the C data protocol is
> > meants for in-memory communication, so different concerns apply
> >
> > If that is how the a particular implementation presents it, that is a
> > weaknesses of the implementation, not the format. The primary use case I
> > was focused on when working on the initial format was communication within
> > the same process. It seems like this is being used as a basis for the
> > introduction of new things when the premise is inconsistent with the
> > intention of the creation. The specific reason we used flatbuffers in the
> > project was to collapse the separation of in-process and out-of-process
> > communication. It means the same thing it does with the Arrow data itself:
> > that a consumer doesn't have to use a particular library to interact with
> > and use the data.
> >
> > It seems like there are two ideas here:
> >
> > 1) How do we make it easier for people to use Arrow?
> > 2) Should we implement a new in memory representation of Arrow that is
> > language specific.
> >
> > I'm entirely in support of number one. If for a particular type of domain,
> > people want an easier way to interact with Arrow, let's make a new library
> > that helps with that. In easy of our current libraries, we do many things
> > to make it easier to work with Arrow. None of those require a change to the
> > core format or are formalized as a new in-memory standard. The in-memory
> > representation of rust or javascript or java objects are implementation
> > details.
> >
> > I'm against number two as it creates a fragmentation problem. Arrow is
> > about having a single canonical format for memory for both metadata and
> > data. Having multiple in-memory formats (especially when some are not
> > language independent) is counter to the goals of the project.
> >
> > Two other, separate comments:
> > 1) I don't understand the idea that we need to change the way Arrow
> > fundamentally works so that people can avoid using a dependency. If the
> > dependency is small, open source and easy to build, people can fork it and
> > include directly if they want to. Let's not violate project principles
> > because DuckDB has a religious perspective on dependencies. If the problem
> > is people have to swallow too large of a pill to do basic things with Arrow
> > in C, let's focus on fixing that (to our definition of ease, not someone
> > else's). If FlatCC solves some those things, great. If we need to build a
> > baby integration library that is more C centric, great. Neither of those
> > things require implementing something at the format level.
> >
> > 2) It seems like we should discuss the data structure problem separately
> > from the reference management concern.
> >
> >
> > On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
> >
> >> hi Antoine,
> >>
> >> On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org> wrote:
> >> >
> >> >
> >> > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> >> > > A couple things:
> >> > >
> >> > > * I think a C protocol / FFI for Arrow array/vectors would be better
> >> > > to have the same "shape" as an assembled array. Note that the C
> >> > > structs here have very nearly the same "shape" as the data structure
> >> > > representing a C++ Array object [1]. The disassembly and reassembly
> >> > > here is substantially simpler than the IPC protocol. A recursive
> >> > > structure in Flatbuffers would make RecordBatch messages much larger,
> >> > > so the flattened / disassembled representation we use for serialized
> >> > > record batches is the correct one
> >> >
> >> > I'm not sure I agree:
> >> >
> >> > - indeed, it's not a coincidence that the ArrowArray struct looks quite
> >> > closely like the C++ ArrayData object :-)  We have good experience with
> >> > that abstraction and it has proven to work quite well
> >> >
> >> > - the IPC format is meant for serialization while the C data protocol is
> >> > meants for in-memory communication, so different concerns apply
> >> >
> >> > - the fact that this makes the layout slightly larger doesn't seem
> >> > important at all; we're not talking about transferring data over the wire
> >> >
> >> > There's also another argument for having a recursive struct: it
> >> > simplifies how the data type is represented, since we can encode each
> >> > child type individually instead of encoding it in the parent's format
> >> > string (same applies for metadata and individual flags).
> >> >
> >>
> >> I was saying something different here. I was making an argument about
> >> why we use the flattened array-of-structs in the IPC protocol. One
> >> reason is that it's a more compact representation. That is not very
> >> important here because this protocol is only for *in-process* (for
> >> languages that have a C FFI facility) rather than *inter-process*
> >> communication.
> >>
> >> I agree also that the type encoding is simple, here, too, since we
> >> aren't having to split the schema and record batch between different
> >> serialized messages. There is some potential waste with having to
> >> populate the type fields multiple times when communicating a sequence
> >> of "chunks" from the same logical dataset.
> >>
> >> > > * The "formal" C protocol having the "assembled" shape means that many
> >> > > minimal Arrow users won't have to implement any separate data
> >> > > structures. They can just use the C struct directly or a slightly
> >> > > wrapped version thereof with some convenience functions.
> >> >
> >> > Yes, but the same applies to the current proposal.
> >> >
> >> > > * I think that requiring building a Flatbuffer for minimal use cases
> >> > > (e.g. communicating simple record batches with primitive types) passes
> >> > > on implementation burden to minimal users.
> >> >
> >> > It certainly does.
> >> >
> >> > > I think the mantra of the C protocol should be the following:
> >> > >
> >> > > * Users of the protocol have to write little to no code to use it. For
> >> > > example, populating an INT32 array should require only a few lines of
> >> > > code
> >> >
> >> > Agreed.  As a sidenote, the spec should have an example of doing this in
> >> > raw C.
> >> >
> >> > Regards
> >> >
> >> > Antoine.
> >>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

As currently designed, it's entirely in-process.  Shared memory with
buffer lifetime handling is taking care of by something like Plasma.

Regards

Antoine.


Le 01/10/2019 à 22:22, Jed Brown a écrit :
> I'd just like to chime in with the use case of in-situ data analysis for
> simulations.  This domain tends to be cautious with dependencies and
> there is a lot of C and Fortran, but the in-situ analysis tools will
> preferably reside in separate processes while sharing memory via shared
> memory (/dev/shm or MPI_Win_allocate_shared).  An in-memory protocol
> that holds raw pointers would be problematic because they are typically
> in different virtual address spaces when shared between processes.  I
> think this is a potential application for a C interface with lean
> dependencies, but it wouldn't be useful if it can't be shared
> out-of-process.

Re: [DISCUSS] C-level in-process array protocol

[Jed Brown <je...@jedbrown.org>]

I'd just like to chime in with the use case of in-situ data analysis for
simulations.  This domain tends to be cautious with dependencies and
there is a lot of C and Fortran, but the in-situ analysis tools will
preferably reside in separate processes while sharing memory via shared
memory (/dev/shm or MPI_Win_allocate_shared).  An in-memory protocol
that holds raw pointers would be problematic because they are typically
in different virtual address spaces when shared between processes.  I
think this is a potential application for a C interface with lean
dependencies, but it wouldn't be useful if it can't be shared
out-of-process.

Jacques Nadeau <ja...@apache.org> writes:

> I disagree with this statement:
>
> - the IPC format is meant for serialization while the C data protocol is
> meants for in-memory communication, so different concerns apply
>
> If that is how the a particular implementation presents it, that is a
> weaknesses of the implementation, not the format. The primary use case I
> was focused on when working on the initial format was communication within
> the same process. It seems like this is being used as a basis for the
> introduction of new things when the premise is inconsistent with the
> intention of the creation. The specific reason we used flatbuffers in the
> project was to collapse the separation of in-process and out-of-process
> communication. It means the same thing it does with the Arrow data itself:
> that a consumer doesn't have to use a particular library to interact with
> and use the data.
>
> It seems like there are two ideas here:
>
> 1) How do we make it easier for people to use Arrow?
> 2) Should we implement a new in memory representation of Arrow that is
> language specific.
>
> I'm entirely in support of number one. If for a particular type of domain,
> people want an easier way to interact with Arrow, let's make a new library
> that helps with that. In easy of our current libraries, we do many things
> to make it easier to work with Arrow. None of those require a change to the
> core format or are formalized as a new in-memory standard. The in-memory
> representation of rust or javascript or java objects are implementation
> details.
>
> I'm against number two as it creates a fragmentation problem. Arrow is
> about having a single canonical format for memory for both metadata and
> data. Having multiple in-memory formats (especially when some are not
> language independent) is counter to the goals of the project.
>
> Two other, separate comments:
> 1) I don't understand the idea that we need to change the way Arrow
> fundamentally works so that people can avoid using a dependency. If the
> dependency is small, open source and easy to build, people can fork it and
> include directly if they want to. Let's not violate project principles
> because DuckDB has a religious perspective on dependencies. If the problem
> is people have to swallow too large of a pill to do basic things with Arrow
> in C, let's focus on fixing that (to our definition of ease, not someone
> else's). If FlatCC solves some those things, great. If we need to build a
> baby integration library that is more C centric, great. Neither of those
> things require implementing something at the format level.
>
> 2) It seems like we should discuss the data structure problem separately
> from the reference management concern.
>
>
> On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:
>
>> hi Antoine,
>>
>> On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org> wrote:
>> >
>> >
>> > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
>> > > A couple things:
>> > >
>> > > * I think a C protocol / FFI for Arrow array/vectors would be better
>> > > to have the same "shape" as an assembled array. Note that the C
>> > > structs here have very nearly the same "shape" as the data structure
>> > > representing a C++ Array object [1]. The disassembly and reassembly
>> > > here is substantially simpler than the IPC protocol. A recursive
>> > > structure in Flatbuffers would make RecordBatch messages much larger,
>> > > so the flattened / disassembled representation we use for serialized
>> > > record batches is the correct one
>> >
>> > I'm not sure I agree:
>> >
>> > - indeed, it's not a coincidence that the ArrowArray struct looks quite
>> > closely like the C++ ArrayData object :-)  We have good experience with
>> > that abstraction and it has proven to work quite well
>> >
>> > - the IPC format is meant for serialization while the C data protocol is
>> > meants for in-memory communication, so different concerns apply
>> >
>> > - the fact that this makes the layout slightly larger doesn't seem
>> > important at all; we're not talking about transferring data over the wire
>> >
>> > There's also another argument for having a recursive struct: it
>> > simplifies how the data type is represented, since we can encode each
>> > child type individually instead of encoding it in the parent's format
>> > string (same applies for metadata and individual flags).
>> >
>>
>> I was saying something different here. I was making an argument about
>> why we use the flattened array-of-structs in the IPC protocol. One
>> reason is that it's a more compact representation. That is not very
>> important here because this protocol is only for *in-process* (for
>> languages that have a C FFI facility) rather than *inter-process*
>> communication.
>>
>> I agree also that the type encoding is simple, here, too, since we
>> aren't having to split the schema and record batch between different
>> serialized messages. There is some potential waste with having to
>> populate the type fields multiple times when communicating a sequence
>> of "chunks" from the same logical dataset.
>>
>> > > * The "formal" C protocol having the "assembled" shape means that many
>> > > minimal Arrow users won't have to implement any separate data
>> > > structures. They can just use the C struct directly or a slightly
>> > > wrapped version thereof with some convenience functions.
>> >
>> > Yes, but the same applies to the current proposal.
>> >
>> > > * I think that requiring building a Flatbuffer for minimal use cases
>> > > (e.g. communicating simple record batches with primitive types) passes
>> > > on implementation burden to minimal users.
>> >
>> > It certainly does.
>> >
>> > > I think the mantra of the C protocol should be the following:
>> > >
>> > > * Users of the protocol have to write little to no code to use it. For
>> > > example, populating an INT32 array should require only a few lines of
>> > > code
>> >
>> > Agreed.  As a sidenote, the spec should have an example of doing this in
>> > raw C.
>> >
>> > Regards
>> >
>> > Antoine.
>>

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

I disagree with this statement:

- the IPC format is meant for serialization while the C data protocol is
meants for in-memory communication, so different concerns apply

If that is how the a particular implementation presents it, that is a
weaknesses of the implementation, not the format. The primary use case I
was focused on when working on the initial format was communication within
the same process. It seems like this is being used as a basis for the
introduction of new things when the premise is inconsistent with the
intention of the creation. The specific reason we used flatbuffers in the
project was to collapse the separation of in-process and out-of-process
communication. It means the same thing it does with the Arrow data itself:
that a consumer doesn't have to use a particular library to interact with
and use the data.

It seems like there are two ideas here:

1) How do we make it easier for people to use Arrow?
2) Should we implement a new in memory representation of Arrow that is
language specific.

I'm entirely in support of number one. If for a particular type of domain,
people want an easier way to interact with Arrow, let's make a new library
that helps with that. In easy of our current libraries, we do many things
to make it easier to work with Arrow. None of those require a change to the
core format or are formalized as a new in-memory standard. The in-memory
representation of rust or javascript or java objects are implementation
details.

I'm against number two as it creates a fragmentation problem. Arrow is
about having a single canonical format for memory for both metadata and
data. Having multiple in-memory formats (especially when some are not
language independent) is counter to the goals of the project.

Two other, separate comments:
1) I don't understand the idea that we need to change the way Arrow
fundamentally works so that people can avoid using a dependency. If the
dependency is small, open source and easy to build, people can fork it and
include directly if they want to. Let's not violate project principles
because DuckDB has a religious perspective on dependencies. If the problem
is people have to swallow too large of a pill to do basic things with Arrow
in C, let's focus on fixing that (to our definition of ease, not someone
else's). If FlatCC solves some those things, great. If we need to build a
baby integration library that is more C centric, great. Neither of those
things require implementing something at the format level.

2) It seems like we should discuss the data structure problem separately
from the reference management concern.


On Tue, Oct 1, 2019 at 5:42 AM Wes McKinney <we...@gmail.com> wrote:

> hi Antoine,
>
> On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > > A couple things:
> > >
> > > * I think a C protocol / FFI for Arrow array/vectors would be better
> > > to have the same "shape" as an assembled array. Note that the C
> > > structs here have very nearly the same "shape" as the data structure
> > > representing a C++ Array object [1]. The disassembly and reassembly
> > > here is substantially simpler than the IPC protocol. A recursive
> > > structure in Flatbuffers would make RecordBatch messages much larger,
> > > so the flattened / disassembled representation we use for serialized
> > > record batches is the correct one
> >
> > I'm not sure I agree:
> >
> > - indeed, it's not a coincidence that the ArrowArray struct looks quite
> > closely like the C++ ArrayData object :-)  We have good experience with
> > that abstraction and it has proven to work quite well
> >
> > - the IPC format is meant for serialization while the C data protocol is
> > meants for in-memory communication, so different concerns apply
> >
> > - the fact that this makes the layout slightly larger doesn't seem
> > important at all; we're not talking about transferring data over the wire
> >
> > There's also another argument for having a recursive struct: it
> > simplifies how the data type is represented, since we can encode each
> > child type individually instead of encoding it in the parent's format
> > string (same applies for metadata and individual flags).
> >
>
> I was saying something different here. I was making an argument about
> why we use the flattened array-of-structs in the IPC protocol. One
> reason is that it's a more compact representation. That is not very
> important here because this protocol is only for *in-process* (for
> languages that have a C FFI facility) rather than *inter-process*
> communication.
>
> I agree also that the type encoding is simple, here, too, since we
> aren't having to split the schema and record batch between different
> serialized messages. There is some potential waste with having to
> populate the type fields multiple times when communicating a sequence
> of "chunks" from the same logical dataset.
>
> > > * The "formal" C protocol having the "assembled" shape means that many
> > > minimal Arrow users won't have to implement any separate data
> > > structures. They can just use the C struct directly or a slightly
> > > wrapped version thereof with some convenience functions.
> >
> > Yes, but the same applies to the current proposal.
> >
> > > * I think that requiring building a Flatbuffer for minimal use cases
> > > (e.g. communicating simple record batches with primitive types) passes
> > > on implementation burden to minimal users.
> >
> > It certainly does.
> >
> > > I think the mantra of the C protocol should be the following:
> > >
> > > * Users of the protocol have to write little to no code to use it. For
> > > example, populating an INT32 array should require only a few lines of
> > > code
> >
> > Agreed.  As a sidenote, the spec should have an example of doing this in
> > raw C.
> >
> > Regards
> >
> > Antoine.
>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

hi Antoine,

On Tue, Oct 1, 2019 at 4:29 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> > A couple things:
> >
> > * I think a C protocol / FFI for Arrow array/vectors would be better
> > to have the same "shape" as an assembled array. Note that the C
> > structs here have very nearly the same "shape" as the data structure
> > representing a C++ Array object [1]. The disassembly and reassembly
> > here is substantially simpler than the IPC protocol. A recursive
> > structure in Flatbuffers would make RecordBatch messages much larger,
> > so the flattened / disassembled representation we use for serialized
> > record batches is the correct one
>
> I'm not sure I agree:
>
> - indeed, it's not a coincidence that the ArrowArray struct looks quite
> closely like the C++ ArrayData object :-)  We have good experience with
> that abstraction and it has proven to work quite well
>
> - the IPC format is meant for serialization while the C data protocol is
> meants for in-memory communication, so different concerns apply
>
> - the fact that this makes the layout slightly larger doesn't seem
> important at all; we're not talking about transferring data over the wire
>
> There's also another argument for having a recursive struct: it
> simplifies how the data type is represented, since we can encode each
> child type individually instead of encoding it in the parent's format
> string (same applies for metadata and individual flags).
>

I was saying something different here. I was making an argument about
why we use the flattened array-of-structs in the IPC protocol. One
reason is that it's a more compact representation. That is not very
important here because this protocol is only for *in-process* (for
languages that have a C FFI facility) rather than *inter-process*
communication.

I agree also that the type encoding is simple, here, too, since we
aren't having to split the schema and record batch between different
serialized messages. There is some potential waste with having to
populate the type fields multiple times when communicating a sequence
of "chunks" from the same logical dataset.

> > * The "formal" C protocol having the "assembled" shape means that many
> > minimal Arrow users won't have to implement any separate data
> > structures. They can just use the C struct directly or a slightly
> > wrapped version thereof with some convenience functions.
>
> Yes, but the same applies to the current proposal.
>
> > * I think that requiring building a Flatbuffer for minimal use cases
> > (e.g. communicating simple record batches with primitive types) passes
> > on implementation burden to minimal users.
>
> It certainly does.
>
> > I think the mantra of the C protocol should be the following:
> >
> > * Users of the protocol have to write little to no code to use it. For
> > example, populating an INT32 array should require only a few lines of
> > code
>
> Agreed.  As a sidenote, the spec should have an example of doing this in
> raw C.
>
> Regards
>
> Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 01/10/2019 à 00:39, Wes McKinney a écrit :
> A couple things:
> 
> * I think a C protocol / FFI for Arrow array/vectors would be better
> to have the same "shape" as an assembled array. Note that the C
> structs here have very nearly the same "shape" as the data structure
> representing a C++ Array object [1]. The disassembly and reassembly
> here is substantially simpler than the IPC protocol. A recursive
> structure in Flatbuffers would make RecordBatch messages much larger,
> so the flattened / disassembled representation we use for serialized
> record batches is the correct one

I'm not sure I agree:

- indeed, it's not a coincidence that the ArrowArray struct looks quite
closely like the C++ ArrayData object :-)  We have good experience with
that abstraction and it has proven to work quite well

- the IPC format is meant for serialization while the C data protocol is
meants for in-memory communication, so different concerns apply

- the fact that this makes the layout slightly larger doesn't seem
important at all; we're not talking about transferring data over the wire

There's also another argument for having a recursive struct: it
simplifies how the data type is represented, since we can encode each
child type individually instead of encoding it in the parent's format
string (same applies for metadata and individual flags).

> * The "formal" C protocol having the "assembled" shape means that many
> minimal Arrow users won't have to implement any separate data
> structures. They can just use the C struct directly or a slightly
> wrapped version thereof with some convenience functions.

Yes, but the same applies to the current proposal.

> * I think that requiring building a Flatbuffer for minimal use cases
> (e.g. communicating simple record batches with primitive types) passes
> on implementation burden to minimal users.

It certainly does.

> I think the mantra of the C protocol should be the following:
> 
> * Users of the protocol have to write little to no code to use it. For
> example, populating an INT32 array should require only a few lines of
> code

Agreed.  As a sidenote, the spec should have an example of doing this in
raw C.

Regards

Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

A couple things:

* I think a C protocol / FFI for Arrow array/vectors would be better
to have the same "shape" as an assembled array. Note that the C
structs here have very nearly the same "shape" as the data structure
representing a C++ Array object [1]. The disassembly and reassembly
here is substantially simpler than the IPC protocol. A recursive
structure in Flatbuffers would make RecordBatch messages much larger,
so the flattened / disassembled representation we use for serialized
record batches is the correct one

* The "formal" C protocol having the "assembled" shape means that many
minimal Arrow users won't have to implement any separate data
structures. They can just use the C struct directly or a slightly
wrapped version thereof with some convenience functions.

* I think that requiring building a Flatbuffer for minimal use cases
(e.g. communicating simple record batches with primitive types) passes
on implementation burden to minimal users.

I think the mantra of the C protocol should be the following:

* Users of the protocol have to write little to no code to use it. For
example, populating an INT32 array should require only a few lines of
code
* The data structure in the protocol is suitable as an in-memory data
structure for recursive assembly of nested structures

I think that having a string miniformat or a pre-parsed type struct
with enum values (along the lines of what Antoine is describing above)
places less burden on downstream users.

[1]: https://github.com/apache/arrow/blob/master/cpp/src/arrow/array.h#L203

On Mon, Sep 30, 2019 at 4:08 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> FlatCC is still a dependency, with generated files etc.
> Perhaps you want to evaluate FlatCC on a schema-like example and see
> what the generated code and compile instructions look like?
>
> I'll point out again that the format string in my proposal uses an
> extremely simple mini-format, that should be parsable very easily by any
> developer, even in raw C:
> https://github.com/apache/arrow/blob/3806fa9ba3ddf95f0d09b865071bf19c5e912756/docs/source/format/CProtocol.rst#data-type-description----format-strings
>
> The parent-child structure in the schema is represented as-is in the
> ArrowArray parent-child relationship, so it doesn't need any encoding.
> Using Flatbuffers for an enum-like field + (at most) a couple parameters
> sounds overkill.
>
> Another possibility would be to replace the format string with
> pre-parsed fields, for example:
>
>   int32_t type;
>   int32_t subtype;      // type-dependent (e.g. unit for temporal types)
>   int32_t type_width;   // for width-parametered types
>   const int8_t* child_ids;   // for unions
>   const char* auxiliary_type_param;  // e.g. timezone for timestamp
>
> The downside is that there are more fields to consider (including two
> optional pointers).
>
> Regards
>
> Antoine.
>
>
> Le 30/09/2019 à 22:48, Ben Kietzman a écrit :
> > FlatCC seems germane: https://github.com/dvidelabs/flatcc
> >
> > It compiles flatbuffer schemas down to (idiomatic?) C
> >
> > Perhaps the schema and batch serialization problems should be solved by
> > storing everything in the flatbuffer format.
> > Then the results of running flatcc plus a few simple helpers can be checked
> > in to provide an accessible C API.
> > With respect to lifetime, Antoine has already done good work on specifying
> > a move only contract which could probably be adapted.
> >
> >
> > On Sun, Sep 29, 2019 at 2:44 PM Antoine Pitrou <an...@python.org> wrote:
> >
> >>
> >> One basic design point is to allow exchanging Arrow data with no
> >> mandatory dependency (the exception is JSON and base64 if you want to
> >> act on metadata - but that's highly optional, and those are extremely
> >> widespread formats).  I'm afraid that Flatbuffers may be a deterrent:
> >> not only it introduces a library, but it requires the use of a compiler
> >> to produce generated code.  It also requires familiarizing with, well,
> >> Flatbuffers :-)
> >>
> >> We can of course discuss this and feel it's not a problem.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> Le 29/09/2019 à 19:47, Wes McKinney a écrit :
> >>> There are two pieces of serialized data needed to communicate a record
> >>> batch from one library to another
> >>>
> >>> * Serialized schema (i.e. what's in Schema.fbs)
> >>> * Serialized "data header", i.e. the "RecordBatch" message in Message.fbs
> >>>
> >>> You _do_ need to use a Flatbuffers library to fully create these
> >>> message types to interact with any existing record batch disassembly /
> >>> reassembly.
> >>>
> >>> I think I'm most concerned about having a new way to serialize
> >>> schemas. We already have JSON-based schema serialization for
> >>> integration test purposes, so one possibility is to standardize that
> >>> and make it a more formalized part of the project specification.
> >>>
> >>> As far as a C protocol, I don't see an especial downside to using the
> >>> Flatbuffers schema to communicate types.
> >>>
> >>> Another thought is to not deviate from the flattened
> >>> Flatbuffers-styled representation but to translate the Flatbuffers
> >>> types into C types: namely a C struct-based version of the
> >>> "RecordBatch" message.
> >>>
> >>> Independent of the means to communicate the two pieces of serialized
> >>> information above (respectively: schemas and record batch field memory
> >>> addresses and field lengths), having a C-based FFI where project can
> >>> drop in a header file containing the ABI they are supposed to
> >>> implement, that seems pretty reasonable to me.
> >>>
> >>> If we don't define a standardized in-memory FFI (whether it uses the
> >>> Flatbuffers objects as inputs/outputs or not) then downstream project
> >>> will devise their own, and that will cause issues long term.
> >>>
> >>> On Sun, Sep 29, 2019 at 2:59 AM Antoine Pitrou <an...@python.org>
> >> wrote:
> >>>>
> >>>>
> >>>> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
> >>>>> * No dependency on Flatbuffers.
> >>>>> * No buffer reassembly (data is already exposed in logical Arrow
> >> format).
> >>>>> * Zero-copy by design.
> >>>>> * Easy to reimplement from scratch.
> >>>>>
> >>>>> I don't see how the flatbuffer pattern for data headers doesn't
> >> accomplish
> >>>>> all of these things. At its definition, is a very simple
> >> representation of
> >>>>> data that could be worked with independently of the flatbuffers
> >> codebase.
> >>>>> It was designed so systems could map directly into that memory without
> >>>>> interacting with a flatbuffers library.
> >>>>>
> >>>>> Specifically the following three structures were designed to already
> >> allow
> >>>>> what I think this proposal is trying to recreate. All three are very
> >> simple
> >>>>> to construct in a direct, non-flatbuffer dependent read/write pattern.
> >>>>
> >>>> Are they?  Personally, I wouldn't know how to do that.  I don't know
> >>>> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
> >>>> it be? if it's portable accross different platforms, then it's probably
> >>>> not compatible with any particular platform's C ABI, or only as a
> >>>> conincidence), how I'm supposed to make use of the "offset" field, or
> >>>> what the lifetime / ownership of all this data is.
> >>>>
> >>>> I may be missing something, but if the answer is that it's easy to
> >>>> reimplement Flatbuffers' encoding without relying on the Flatbuffers
> >>>> project's source code, I'm a bit skeptical.
> >>>>
> >>>> Regards
> >>>>
> >>>> Antoine.
> >>>>
> >>>>
> >>>>>
> >>>>> struct FieldNode {
> >>>>>   length: long;
> >>>>>   null_count: long;
> >>>>> }
> >>>>>
> >>>>> struct Buffer {
> >>>>>   offset: long;
> >>>>>   length: long;
> >>>>> }
> >>>>>
> >>>>> table RecordBatch {
> >>>>>   length: long;
> >>>>>   nodes: [FieldNode];
> >>>>>   buffers: [Buffer];
> >>>>> }
> >>>>>
> >>>>> On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org>
> >> wrote:
> >>>>>
> >>>>>> I'm not clear on why we need to introduce something beyond what
> >>>>>> flatbuffers already provides. Can someone explain that to me? I'm not
> >>>>>> really a fan of introducing a second representation of the same data
> >> (as I
> >>>>>> understand it).
> >>>>>>
> >>>>>> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> This is helpful, I will leave some comments on the proposal when I
> >>>>>>> can, sometime in the next week.
> >>>>>>>
> >>>>>>> I agree that it would likely be opening a can of worms to create a
> >>>>>>> semantic mapping between a generalized type grammar and Arrow's
> >>>>>>> specific logical types defined in Schema.fbs. If we go down this
> >>>>>>> route, we should probably utilize the simplest possible grammar that
> >>>>>>> is capable of encoding the Type Flatbuffers union values.
> >>>>>>>
> >>>>>>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've posted a draft specification PR here, this should help orient
> >> the
> >>>>>>>> discussion a bit:
> >>>>>>>> https://github.com/apache/arrow/pull/5442
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>>
> >>>>>>>> Antoine.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, 18 Sep 2019 19:52:38 +0200
> >>>>>>>> Antoine Pitrou <an...@python.org> wrote:
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> One thing that was discussed in the sync call is the ability to
> >> easily
> >>>>>>>>> pass arrays at runtime between Arrow implementations or
> >>>>>>> Arrow-supporting
> >>>>>>>>> libraries in the same process, without bearing the cost of linking
> >> to
> >>>>>>>>> e.g. the C++ Arrow library.
> >>>>>>>>>
> >>>>>>>>> (for example: "Duckdb wants to provide an option to return Arrow
> >> data
> >>>>>>> of
> >>>>>>>>> result sets, but they don't like having Arrow as a dependency")
> >>>>>>>>>
> >>>>>>>>> One possibility would be to define a C-level protocol similar in
> >>>>>>> spirit
> >>>>>>>>> to the Python buffer protocol, which some people may be familiar
> >> with
> >>>>>>> (*).
> >>>>>>>>>
> >>>>>>>>> The basic idea is to define a simple C struct, which is ABI-stable
> >> and
> >>>>>>>>> describes an Arrow away adequately.  The struct can be
> >>>>>>> stack-allocated.
> >>>>>>>>> Its definition can also be copied in another project (or interfaced
> >>>>>>> with
> >>>>>>>>> using a C FFI layer, depending on the language).
> >>>>>>>>>
> >>>>>>>>> There is no formal proposal, this message is meant to stir the
> >>>>>>> discussion.
> >>>>>>>>>
> >>>>>>>>> Issues to work out:
> >>>>>>>>>
> >>>>>>>>> * Memory lifetime issues: where Python simply associates the
> >> Py_buffer
> >>>>>>>>> with a PyObject owner (a garbage-collected Python object), we need
> >>>>>>>>> another means to control lifetime of pointed areas.  One simple
> >>>>>>>>> possibility is to include a destructor function pointer in the
> >>>>>>> protocol
> >>>>>>>>> struct.
> >>>>>>>>>
> >>>>>>>>> * Arrow type representation.  We probably need some kind of
> >> "format"
> >>>>>>>>> mini-language to represent Arrow types, so that a type can be
> >>>>>>> described
> >>>>>>>>> using a `const char*`.  Ideally, primitives types at least should
> >> be
> >>>>>>>>> trivially parsable.  We may take inspiration from Python here
> >>>>>>> (`struct`
> >>>>>>>>> module format characters, PEP 3118 format additions).
> >>>>>>>>>
> >>>>>>>>> Example C struct definition (not a formal proposal!):
> >>>>>>>>>
> >>>>>>>>> struct ArrowBuffer {
> >>>>>>>>>   void* data;
> >>>>>>>>>   int64_t nbytes;
> >>>>>>>>>   // Called by the consumer when it doesn't need the buffer anymore
> >>>>>>>>>   void (*release)(struct ArrowBuffer*);
> >>>>>>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>>>>>   void* user_data;
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> struct ArrowArray {
> >>>>>>>>>   // Type description
> >>>>>>>>>   const char* format;
> >>>>>>>>>   // Data description
> >>>>>>>>>   int64_t length;
> >>>>>>>>>   int64_t null_count;
> >>>>>>>>>   int64_t n_buffers;
> >>>>>>>>>   // Note: this pointers are probably owned by the ArrowArray
> >> struct
> >>>>>>>>>   // and will be released and free()ed by the release callback.
> >>>>>>>>>   struct BufferDescriptor* buffers;
> >>>>>>>>>   struct ArrowDescriptor* dictionary;
> >>>>>>>>>   // Called by the consumer when it doesn't need the array anymore
> >>>>>>>>>   void (*release)(struct ArrowArrayDescriptor*);
> >>>>>>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>>>>>   void* user_data;
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>>
> >>>>>>>>> (*) For the record, the reference for the Python buffer protocol:
> >>>>>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> >>>>>>>>> and its C struct definition:
> >>>>>>>>>
> >>>>>>>
> >> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >>>>>>>>>
> >>>>>>>>> Regards
> >>>>>>>>>
> >>>>>>>>> Antoine.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
> >

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

FlatCC is still a dependency, with generated files etc.
Perhaps you want to evaluate FlatCC on a schema-like example and see
what the generated code and compile instructions look like?

I'll point out again that the format string in my proposal uses an
extremely simple mini-format, that should be parsable very easily by any
developer, even in raw C:
https://github.com/apache/arrow/blob/3806fa9ba3ddf95f0d09b865071bf19c5e912756/docs/source/format/CProtocol.rst#data-type-description----format-strings

The parent-child structure in the schema is represented as-is in the
ArrowArray parent-child relationship, so it doesn't need any encoding.
Using Flatbuffers for an enum-like field + (at most) a couple parameters
sounds overkill.

Another possibility would be to replace the format string with
pre-parsed fields, for example:

  int32_t type;
  int32_t subtype;      // type-dependent (e.g. unit for temporal types)
  int32_t type_width;   // for width-parametered types
  const int8_t* child_ids;   // for unions
  const char* auxiliary_type_param;  // e.g. timezone for timestamp

The downside is that there are more fields to consider (including two
optional pointers).

Regards

Antoine.


Le 30/09/2019 à 22:48, Ben Kietzman a écrit :
> FlatCC seems germane: https://github.com/dvidelabs/flatcc
> 
> It compiles flatbuffer schemas down to (idiomatic?) C
> 
> Perhaps the schema and batch serialization problems should be solved by
> storing everything in the flatbuffer format.
> Then the results of running flatcc plus a few simple helpers can be checked
> in to provide an accessible C API.
> With respect to lifetime, Antoine has already done good work on specifying
> a move only contract which could probably be adapted.
> 
> 
> On Sun, Sep 29, 2019 at 2:44 PM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> One basic design point is to allow exchanging Arrow data with no
>> mandatory dependency (the exception is JSON and base64 if you want to
>> act on metadata - but that's highly optional, and those are extremely
>> widespread formats).  I'm afraid that Flatbuffers may be a deterrent:
>> not only it introduces a library, but it requires the use of a compiler
>> to produce generated code.  It also requires familiarizing with, well,
>> Flatbuffers :-)
>>
>> We can of course discuss this and feel it's not a problem.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 29/09/2019 à 19:47, Wes McKinney a écrit :
>>> There are two pieces of serialized data needed to communicate a record
>>> batch from one library to another
>>>
>>> * Serialized schema (i.e. what's in Schema.fbs)
>>> * Serialized "data header", i.e. the "RecordBatch" message in Message.fbs
>>>
>>> You _do_ need to use a Flatbuffers library to fully create these
>>> message types to interact with any existing record batch disassembly /
>>> reassembly.
>>>
>>> I think I'm most concerned about having a new way to serialize
>>> schemas. We already have JSON-based schema serialization for
>>> integration test purposes, so one possibility is to standardize that
>>> and make it a more formalized part of the project specification.
>>>
>>> As far as a C protocol, I don't see an especial downside to using the
>>> Flatbuffers schema to communicate types.
>>>
>>> Another thought is to not deviate from the flattened
>>> Flatbuffers-styled representation but to translate the Flatbuffers
>>> types into C types: namely a C struct-based version of the
>>> "RecordBatch" message.
>>>
>>> Independent of the means to communicate the two pieces of serialized
>>> information above (respectively: schemas and record batch field memory
>>> addresses and field lengths), having a C-based FFI where project can
>>> drop in a header file containing the ABI they are supposed to
>>> implement, that seems pretty reasonable to me.
>>>
>>> If we don't define a standardized in-memory FFI (whether it uses the
>>> Flatbuffers objects as inputs/outputs or not) then downstream project
>>> will devise their own, and that will cause issues long term.
>>>
>>> On Sun, Sep 29, 2019 at 2:59 AM Antoine Pitrou <an...@python.org>
>> wrote:
>>>>
>>>>
>>>> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
>>>>> * No dependency on Flatbuffers.
>>>>> * No buffer reassembly (data is already exposed in logical Arrow
>> format).
>>>>> * Zero-copy by design.
>>>>> * Easy to reimplement from scratch.
>>>>>
>>>>> I don't see how the flatbuffer pattern for data headers doesn't
>> accomplish
>>>>> all of these things. At its definition, is a very simple
>> representation of
>>>>> data that could be worked with independently of the flatbuffers
>> codebase.
>>>>> It was designed so systems could map directly into that memory without
>>>>> interacting with a flatbuffers library.
>>>>>
>>>>> Specifically the following three structures were designed to already
>> allow
>>>>> what I think this proposal is trying to recreate. All three are very
>> simple
>>>>> to construct in a direct, non-flatbuffer dependent read/write pattern.
>>>>
>>>> Are they?  Personally, I wouldn't know how to do that.  I don't know
>>>> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
>>>> it be? if it's portable accross different platforms, then it's probably
>>>> not compatible with any particular platform's C ABI, or only as a
>>>> conincidence), how I'm supposed to make use of the "offset" field, or
>>>> what the lifetime / ownership of all this data is.
>>>>
>>>> I may be missing something, but if the answer is that it's easy to
>>>> reimplement Flatbuffers' encoding without relying on the Flatbuffers
>>>> project's source code, I'm a bit skeptical.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>>>
>>>>> struct FieldNode {
>>>>>   length: long;
>>>>>   null_count: long;
>>>>> }
>>>>>
>>>>> struct Buffer {
>>>>>   offset: long;
>>>>>   length: long;
>>>>> }
>>>>>
>>>>> table RecordBatch {
>>>>>   length: long;
>>>>>   nodes: [FieldNode];
>>>>>   buffers: [Buffer];
>>>>> }
>>>>>
>>>>> On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org>
>> wrote:
>>>>>
>>>>>> I'm not clear on why we need to introduce something beyond what
>>>>>> flatbuffers already provides. Can someone explain that to me? I'm not
>>>>>> really a fan of introducing a second representation of the same data
>> (as I
>>>>>> understand it).
>>>>>>
>>>>>> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com>
>> wrote:
>>>>>>
>>>>>>> This is helpful, I will leave some comments on the proposal when I
>>>>>>> can, sometime in the next week.
>>>>>>>
>>>>>>> I agree that it would likely be opening a can of worms to create a
>>>>>>> semantic mapping between a generalized type grammar and Arrow's
>>>>>>> specific logical types defined in Schema.fbs. If we go down this
>>>>>>> route, we should probably utilize the simplest possible grammar that
>>>>>>> is capable of encoding the Type Flatbuffers union values.
>>>>>>>
>>>>>>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I've posted a draft specification PR here, this should help orient
>> the
>>>>>>>> discussion a bit:
>>>>>>>> https://github.com/apache/arrow/pull/5442
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Antoine.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 18 Sep 2019 19:52:38 +0200
>>>>>>>> Antoine Pitrou <an...@python.org> wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> One thing that was discussed in the sync call is the ability to
>> easily
>>>>>>>>> pass arrays at runtime between Arrow implementations or
>>>>>>> Arrow-supporting
>>>>>>>>> libraries in the same process, without bearing the cost of linking
>> to
>>>>>>>>> e.g. the C++ Arrow library.
>>>>>>>>>
>>>>>>>>> (for example: "Duckdb wants to provide an option to return Arrow
>> data
>>>>>>> of
>>>>>>>>> result sets, but they don't like having Arrow as a dependency")
>>>>>>>>>
>>>>>>>>> One possibility would be to define a C-level protocol similar in
>>>>>>> spirit
>>>>>>>>> to the Python buffer protocol, which some people may be familiar
>> with
>>>>>>> (*).
>>>>>>>>>
>>>>>>>>> The basic idea is to define a simple C struct, which is ABI-stable
>> and
>>>>>>>>> describes an Arrow away adequately.  The struct can be
>>>>>>> stack-allocated.
>>>>>>>>> Its definition can also be copied in another project (or interfaced
>>>>>>> with
>>>>>>>>> using a C FFI layer, depending on the language).
>>>>>>>>>
>>>>>>>>> There is no formal proposal, this message is meant to stir the
>>>>>>> discussion.
>>>>>>>>>
>>>>>>>>> Issues to work out:
>>>>>>>>>
>>>>>>>>> * Memory lifetime issues: where Python simply associates the
>> Py_buffer
>>>>>>>>> with a PyObject owner (a garbage-collected Python object), we need
>>>>>>>>> another means to control lifetime of pointed areas.  One simple
>>>>>>>>> possibility is to include a destructor function pointer in the
>>>>>>> protocol
>>>>>>>>> struct.
>>>>>>>>>
>>>>>>>>> * Arrow type representation.  We probably need some kind of
>> "format"
>>>>>>>>> mini-language to represent Arrow types, so that a type can be
>>>>>>> described
>>>>>>>>> using a `const char*`.  Ideally, primitives types at least should
>> be
>>>>>>>>> trivially parsable.  We may take inspiration from Python here
>>>>>>> (`struct`
>>>>>>>>> module format characters, PEP 3118 format additions).
>>>>>>>>>
>>>>>>>>> Example C struct definition (not a formal proposal!):
>>>>>>>>>
>>>>>>>>> struct ArrowBuffer {
>>>>>>>>>   void* data;
>>>>>>>>>   int64_t nbytes;
>>>>>>>>>   // Called by the consumer when it doesn't need the buffer anymore
>>>>>>>>>   void (*release)(struct ArrowBuffer*);
>>>>>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>>>>>   void* user_data;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> struct ArrowArray {
>>>>>>>>>   // Type description
>>>>>>>>>   const char* format;
>>>>>>>>>   // Data description
>>>>>>>>>   int64_t length;
>>>>>>>>>   int64_t null_count;
>>>>>>>>>   int64_t n_buffers;
>>>>>>>>>   // Note: this pointers are probably owned by the ArrowArray
>> struct
>>>>>>>>>   // and will be released and free()ed by the release callback.
>>>>>>>>>   struct BufferDescriptor* buffers;
>>>>>>>>>   struct ArrowDescriptor* dictionary;
>>>>>>>>>   // Called by the consumer when it doesn't need the array anymore
>>>>>>>>>   void (*release)(struct ArrowArrayDescriptor*);
>>>>>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>>>>>   void* user_data;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> (*) For the record, the reference for the Python buffer protocol:
>>>>>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>>>>>>>>> and its C struct definition:
>>>>>>>>>
>>>>>>>
>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> Antoine.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
> 

Re: [DISCUSS] C-level in-process array protocol

[Ben Kietzman <be...@rstudio.com>]

FlatCC seems germane: https://github.com/dvidelabs/flatcc

It compiles flatbuffer schemas down to (idiomatic?) C

Perhaps the schema and batch serialization problems should be solved by
storing everything in the flatbuffer format.
Then the results of running flatcc plus a few simple helpers can be checked
in to provide an accessible C API.
With respect to lifetime, Antoine has already done good work on specifying
a move only contract which could probably be adapted.


On Sun, Sep 29, 2019 at 2:44 PM Antoine Pitrou <an...@python.org> wrote:

>
> One basic design point is to allow exchanging Arrow data with no
> mandatory dependency (the exception is JSON and base64 if you want to
> act on metadata - but that's highly optional, and those are extremely
> widespread formats).  I'm afraid that Flatbuffers may be a deterrent:
> not only it introduces a library, but it requires the use of a compiler
> to produce generated code.  It also requires familiarizing with, well,
> Flatbuffers :-)
>
> We can of course discuss this and feel it's not a problem.
>
> Regards
>
> Antoine.
>
>
> Le 29/09/2019 à 19:47, Wes McKinney a écrit :
> > There are two pieces of serialized data needed to communicate a record
> > batch from one library to another
> >
> > * Serialized schema (i.e. what's in Schema.fbs)
> > * Serialized "data header", i.e. the "RecordBatch" message in Message.fbs
> >
> > You _do_ need to use a Flatbuffers library to fully create these
> > message types to interact with any existing record batch disassembly /
> > reassembly.
> >
> > I think I'm most concerned about having a new way to serialize
> > schemas. We already have JSON-based schema serialization for
> > integration test purposes, so one possibility is to standardize that
> > and make it a more formalized part of the project specification.
> >
> > As far as a C protocol, I don't see an especial downside to using the
> > Flatbuffers schema to communicate types.
> >
> > Another thought is to not deviate from the flattened
> > Flatbuffers-styled representation but to translate the Flatbuffers
> > types into C types: namely a C struct-based version of the
> > "RecordBatch" message.
> >
> > Independent of the means to communicate the two pieces of serialized
> > information above (respectively: schemas and record batch field memory
> > addresses and field lengths), having a C-based FFI where project can
> > drop in a header file containing the ABI they are supposed to
> > implement, that seems pretty reasonable to me.
> >
> > If we don't define a standardized in-memory FFI (whether it uses the
> > Flatbuffers objects as inputs/outputs or not) then downstream project
> > will devise their own, and that will cause issues long term.
> >
> > On Sun, Sep 29, 2019 at 2:59 AM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >>
> >> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
> >>> * No dependency on Flatbuffers.
> >>> * No buffer reassembly (data is already exposed in logical Arrow
> format).
> >>> * Zero-copy by design.
> >>> * Easy to reimplement from scratch.
> >>>
> >>> I don't see how the flatbuffer pattern for data headers doesn't
> accomplish
> >>> all of these things. At its definition, is a very simple
> representation of
> >>> data that could be worked with independently of the flatbuffers
> codebase.
> >>> It was designed so systems could map directly into that memory without
> >>> interacting with a flatbuffers library.
> >>>
> >>> Specifically the following three structures were designed to already
> allow
> >>> what I think this proposal is trying to recreate. All three are very
> simple
> >>> to construct in a direct, non-flatbuffer dependent read/write pattern.
> >>
> >> Are they?  Personally, I wouldn't know how to do that.  I don't know
> >> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
> >> it be? if it's portable accross different platforms, then it's probably
> >> not compatible with any particular platform's C ABI, or only as a
> >> conincidence), how I'm supposed to make use of the "offset" field, or
> >> what the lifetime / ownership of all this data is.
> >>
> >> I may be missing something, but if the answer is that it's easy to
> >> reimplement Flatbuffers' encoding without relying on the Flatbuffers
> >> project's source code, I'm a bit skeptical.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>>
> >>> struct FieldNode {
> >>>   length: long;
> >>>   null_count: long;
> >>> }
> >>>
> >>> struct Buffer {
> >>>   offset: long;
> >>>   length: long;
> >>> }
> >>>
> >>> table RecordBatch {
> >>>   length: long;
> >>>   nodes: [FieldNode];
> >>>   buffers: [Buffer];
> >>> }
> >>>
> >>> On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org>
> wrote:
> >>>
> >>>> I'm not clear on why we need to introduce something beyond what
> >>>> flatbuffers already provides. Can someone explain that to me? I'm not
> >>>> really a fan of introducing a second representation of the same data
> (as I
> >>>> understand it).
> >>>>
> >>>> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com>
> wrote:
> >>>>
> >>>>> This is helpful, I will leave some comments on the proposal when I
> >>>>> can, sometime in the next week.
> >>>>>
> >>>>> I agree that it would likely be opening a can of worms to create a
> >>>>> semantic mapping between a generalized type grammar and Arrow's
> >>>>> specific logical types defined in Schema.fbs. If we go down this
> >>>>> route, we should probably utilize the simplest possible grammar that
> >>>>> is capable of encoding the Type Flatbuffers union values.
> >>>>>
> >>>>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
> >>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> I've posted a draft specification PR here, this should help orient
> the
> >>>>>> discussion a bit:
> >>>>>> https://github.com/apache/arrow/pull/5442
> >>>>>>
> >>>>>> Regards
> >>>>>>
> >>>>>> Antoine.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 18 Sep 2019 19:52:38 +0200
> >>>>>> Antoine Pitrou <an...@python.org> wrote:
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> One thing that was discussed in the sync call is the ability to
> easily
> >>>>>>> pass arrays at runtime between Arrow implementations or
> >>>>> Arrow-supporting
> >>>>>>> libraries in the same process, without bearing the cost of linking
> to
> >>>>>>> e.g. the C++ Arrow library.
> >>>>>>>
> >>>>>>> (for example: "Duckdb wants to provide an option to return Arrow
> data
> >>>>> of
> >>>>>>> result sets, but they don't like having Arrow as a dependency")
> >>>>>>>
> >>>>>>> One possibility would be to define a C-level protocol similar in
> >>>>> spirit
> >>>>>>> to the Python buffer protocol, which some people may be familiar
> with
> >>>>> (*).
> >>>>>>>
> >>>>>>> The basic idea is to define a simple C struct, which is ABI-stable
> and
> >>>>>>> describes an Arrow away adequately.  The struct can be
> >>>>> stack-allocated.
> >>>>>>> Its definition can also be copied in another project (or interfaced
> >>>>> with
> >>>>>>> using a C FFI layer, depending on the language).
> >>>>>>>
> >>>>>>> There is no formal proposal, this message is meant to stir the
> >>>>> discussion.
> >>>>>>>
> >>>>>>> Issues to work out:
> >>>>>>>
> >>>>>>> * Memory lifetime issues: where Python simply associates the
> Py_buffer
> >>>>>>> with a PyObject owner (a garbage-collected Python object), we need
> >>>>>>> another means to control lifetime of pointed areas.  One simple
> >>>>>>> possibility is to include a destructor function pointer in the
> >>>>> protocol
> >>>>>>> struct.
> >>>>>>>
> >>>>>>> * Arrow type representation.  We probably need some kind of
> "format"
> >>>>>>> mini-language to represent Arrow types, so that a type can be
> >>>>> described
> >>>>>>> using a `const char*`.  Ideally, primitives types at least should
> be
> >>>>>>> trivially parsable.  We may take inspiration from Python here
> >>>>> (`struct`
> >>>>>>> module format characters, PEP 3118 format additions).
> >>>>>>>
> >>>>>>> Example C struct definition (not a formal proposal!):
> >>>>>>>
> >>>>>>> struct ArrowBuffer {
> >>>>>>>   void* data;
> >>>>>>>   int64_t nbytes;
> >>>>>>>   // Called by the consumer when it doesn't need the buffer anymore
> >>>>>>>   void (*release)(struct ArrowBuffer*);
> >>>>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>>>   void* user_data;
> >>>>>>> };
> >>>>>>>
> >>>>>>> struct ArrowArray {
> >>>>>>>   // Type description
> >>>>>>>   const char* format;
> >>>>>>>   // Data description
> >>>>>>>   int64_t length;
> >>>>>>>   int64_t null_count;
> >>>>>>>   int64_t n_buffers;
> >>>>>>>   // Note: this pointers are probably owned by the ArrowArray
> struct
> >>>>>>>   // and will be released and free()ed by the release callback.
> >>>>>>>   struct BufferDescriptor* buffers;
> >>>>>>>   struct ArrowDescriptor* dictionary;
> >>>>>>>   // Called by the consumer when it doesn't need the array anymore
> >>>>>>>   void (*release)(struct ArrowArrayDescriptor*);
> >>>>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>>>   void* user_data;
> >>>>>>> };
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> (*) For the record, the reference for the Python buffer protocol:
> >>>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> >>>>>>> and its C struct definition:
> >>>>>>>
> >>>>>
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >>>>>>>
> >>>>>>> Regards
> >>>>>>>
> >>>>>>> Antoine.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

One basic design point is to allow exchanging Arrow data with no
mandatory dependency (the exception is JSON and base64 if you want to
act on metadata - but that's highly optional, and those are extremely
widespread formats).  I'm afraid that Flatbuffers may be a deterrent:
not only it introduces a library, but it requires the use of a compiler
to produce generated code.  It also requires familiarizing with, well,
Flatbuffers :-)

We can of course discuss this and feel it's not a problem.

Regards

Antoine.


Le 29/09/2019 à 19:47, Wes McKinney a écrit :
> There are two pieces of serialized data needed to communicate a record
> batch from one library to another
> 
> * Serialized schema (i.e. what's in Schema.fbs)
> * Serialized "data header", i.e. the "RecordBatch" message in Message.fbs
> 
> You _do_ need to use a Flatbuffers library to fully create these
> message types to interact with any existing record batch disassembly /
> reassembly.
> 
> I think I'm most concerned about having a new way to serialize
> schemas. We already have JSON-based schema serialization for
> integration test purposes, so one possibility is to standardize that
> and make it a more formalized part of the project specification.
> 
> As far as a C protocol, I don't see an especial downside to using the
> Flatbuffers schema to communicate types.
> 
> Another thought is to not deviate from the flattened
> Flatbuffers-styled representation but to translate the Flatbuffers
> types into C types: namely a C struct-based version of the
> "RecordBatch" message.
> 
> Independent of the means to communicate the two pieces of serialized
> information above (respectively: schemas and record batch field memory
> addresses and field lengths), having a C-based FFI where project can
> drop in a header file containing the ABI they are supposed to
> implement, that seems pretty reasonable to me.
> 
> If we don't define a standardized in-memory FFI (whether it uses the
> Flatbuffers objects as inputs/outputs or not) then downstream project
> will devise their own, and that will cause issues long term.
> 
> On Sun, Sep 29, 2019 at 2:59 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
>>> * No dependency on Flatbuffers.
>>> * No buffer reassembly (data is already exposed in logical Arrow format).
>>> * Zero-copy by design.
>>> * Easy to reimplement from scratch.
>>>
>>> I don't see how the flatbuffer pattern for data headers doesn't accomplish
>>> all of these things. At its definition, is a very simple representation of
>>> data that could be worked with independently of the flatbuffers codebase.
>>> It was designed so systems could map directly into that memory without
>>> interacting with a flatbuffers library.
>>>
>>> Specifically the following three structures were designed to already allow
>>> what I think this proposal is trying to recreate. All three are very simple
>>> to construct in a direct, non-flatbuffer dependent read/write pattern.
>>
>> Are they?  Personally, I wouldn't know how to do that.  I don't know
>> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
>> it be? if it's portable accross different platforms, then it's probably
>> not compatible with any particular platform's C ABI, or only as a
>> conincidence), how I'm supposed to make use of the "offset" field, or
>> what the lifetime / ownership of all this data is.
>>
>> I may be missing something, but if the answer is that it's easy to
>> reimplement Flatbuffers' encoding without relying on the Flatbuffers
>> project's source code, I'm a bit skeptical.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>>
>>> struct FieldNode {
>>>   length: long;
>>>   null_count: long;
>>> }
>>>
>>> struct Buffer {
>>>   offset: long;
>>>   length: long;
>>> }
>>>
>>> table RecordBatch {
>>>   length: long;
>>>   nodes: [FieldNode];
>>>   buffers: [Buffer];
>>> }
>>>
>>> On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org> wrote:
>>>
>>>> I'm not clear on why we need to introduce something beyond what
>>>> flatbuffers already provides. Can someone explain that to me? I'm not
>>>> really a fan of introducing a second representation of the same data (as I
>>>> understand it).
>>>>
>>>> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>>> This is helpful, I will leave some comments on the proposal when I
>>>>> can, sometime in the next week.
>>>>>
>>>>> I agree that it would likely be opening a can of worms to create a
>>>>> semantic mapping between a generalized type grammar and Arrow's
>>>>> specific logical types defined in Schema.fbs. If we go down this
>>>>> route, we should probably utilize the simplest possible grammar that
>>>>> is capable of encoding the Type Flatbuffers union values.
>>>>>
>>>>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> I've posted a draft specification PR here, this should help orient the
>>>>>> discussion a bit:
>>>>>> https://github.com/apache/arrow/pull/5442
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Antoine.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 18 Sep 2019 19:52:38 +0200
>>>>>> Antoine Pitrou <an...@python.org> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> One thing that was discussed in the sync call is the ability to easily
>>>>>>> pass arrays at runtime between Arrow implementations or
>>>>> Arrow-supporting
>>>>>>> libraries in the same process, without bearing the cost of linking to
>>>>>>> e.g. the C++ Arrow library.
>>>>>>>
>>>>>>> (for example: "Duckdb wants to provide an option to return Arrow data
>>>>> of
>>>>>>> result sets, but they don't like having Arrow as a dependency")
>>>>>>>
>>>>>>> One possibility would be to define a C-level protocol similar in
>>>>> spirit
>>>>>>> to the Python buffer protocol, which some people may be familiar with
>>>>> (*).
>>>>>>>
>>>>>>> The basic idea is to define a simple C struct, which is ABI-stable and
>>>>>>> describes an Arrow away adequately.  The struct can be
>>>>> stack-allocated.
>>>>>>> Its definition can also be copied in another project (or interfaced
>>>>> with
>>>>>>> using a C FFI layer, depending on the language).
>>>>>>>
>>>>>>> There is no formal proposal, this message is meant to stir the
>>>>> discussion.
>>>>>>>
>>>>>>> Issues to work out:
>>>>>>>
>>>>>>> * Memory lifetime issues: where Python simply associates the Py_buffer
>>>>>>> with a PyObject owner (a garbage-collected Python object), we need
>>>>>>> another means to control lifetime of pointed areas.  One simple
>>>>>>> possibility is to include a destructor function pointer in the
>>>>> protocol
>>>>>>> struct.
>>>>>>>
>>>>>>> * Arrow type representation.  We probably need some kind of "format"
>>>>>>> mini-language to represent Arrow types, so that a type can be
>>>>> described
>>>>>>> using a `const char*`.  Ideally, primitives types at least should be
>>>>>>> trivially parsable.  We may take inspiration from Python here
>>>>> (`struct`
>>>>>>> module format characters, PEP 3118 format additions).
>>>>>>>
>>>>>>> Example C struct definition (not a formal proposal!):
>>>>>>>
>>>>>>> struct ArrowBuffer {
>>>>>>>   void* data;
>>>>>>>   int64_t nbytes;
>>>>>>>   // Called by the consumer when it doesn't need the buffer anymore
>>>>>>>   void (*release)(struct ArrowBuffer*);
>>>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>>>   void* user_data;
>>>>>>> };
>>>>>>>
>>>>>>> struct ArrowArray {
>>>>>>>   // Type description
>>>>>>>   const char* format;
>>>>>>>   // Data description
>>>>>>>   int64_t length;
>>>>>>>   int64_t null_count;
>>>>>>>   int64_t n_buffers;
>>>>>>>   // Note: this pointers are probably owned by the ArrowArray struct
>>>>>>>   // and will be released and free()ed by the release callback.
>>>>>>>   struct BufferDescriptor* buffers;
>>>>>>>   struct ArrowDescriptor* dictionary;
>>>>>>>   // Called by the consumer when it doesn't need the array anymore
>>>>>>>   void (*release)(struct ArrowArrayDescriptor*);
>>>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>>>   void* user_data;
>>>>>>> };
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> (*) For the record, the reference for the Python buffer protocol:
>>>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>>>>>>> and its C struct definition:
>>>>>>>
>>>>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Antoine.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

There are two pieces of serialized data needed to communicate a record
batch from one library to another

* Serialized schema (i.e. what's in Schema.fbs)
* Serialized "data header", i.e. the "RecordBatch" message in Message.fbs

You _do_ need to use a Flatbuffers library to fully create these
message types to interact with any existing record batch disassembly /
reassembly.

I think I'm most concerned about having a new way to serialize
schemas. We already have JSON-based schema serialization for
integration test purposes, so one possibility is to standardize that
and make it a more formalized part of the project specification.

As far as a C protocol, I don't see an especial downside to using the
Flatbuffers schema to communicate types.

Another thought is to not deviate from the flattened
Flatbuffers-styled representation but to translate the Flatbuffers
types into C types: namely a C struct-based version of the
"RecordBatch" message.

Independent of the means to communicate the two pieces of serialized
information above (respectively: schemas and record batch field memory
addresses and field lengths), having a C-based FFI where project can
drop in a header file containing the ABI they are supposed to
implement, that seems pretty reasonable to me.

If we don't define a standardized in-memory FFI (whether it uses the
Flatbuffers objects as inputs/outputs or not) then downstream project
will devise their own, and that will cause issues long term.

On Sun, Sep 29, 2019 at 2:59 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
> > * No dependency on Flatbuffers.
> > * No buffer reassembly (data is already exposed in logical Arrow format).
> > * Zero-copy by design.
> > * Easy to reimplement from scratch.
> >
> > I don't see how the flatbuffer pattern for data headers doesn't accomplish
> > all of these things. At its definition, is a very simple representation of
> > data that could be worked with independently of the flatbuffers codebase.
> > It was designed so systems could map directly into that memory without
> > interacting with a flatbuffers library.
> >
> > Specifically the following three structures were designed to already allow
> > what I think this proposal is trying to recreate. All three are very simple
> > to construct in a direct, non-flatbuffer dependent read/write pattern.
>
> Are they?  Personally, I wouldn't know how to do that.  I don't know
> which encoding Flatbuffers use, whether it's C ABI-compatible (how could
> it be? if it's portable accross different platforms, then it's probably
> not compatible with any particular platform's C ABI, or only as a
> conincidence), how I'm supposed to make use of the "offset" field, or
> what the lifetime / ownership of all this data is.
>
> I may be missing something, but if the answer is that it's easy to
> reimplement Flatbuffers' encoding without relying on the Flatbuffers
> project's source code, I'm a bit skeptical.
>
> Regards
>
> Antoine.
>
>
> >
> > struct FieldNode {
> >   length: long;
> >   null_count: long;
> > }
> >
> > struct Buffer {
> >   offset: long;
> >   length: long;
> > }
> >
> > table RecordBatch {
> >   length: long;
> >   nodes: [FieldNode];
> >   buffers: [Buffer];
> > }
> >
> > On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org> wrote:
> >
> >> I'm not clear on why we need to introduce something beyond what
> >> flatbuffers already provides. Can someone explain that to me? I'm not
> >> really a fan of introducing a second representation of the same data (as I
> >> understand it).
> >>
> >> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >>> This is helpful, I will leave some comments on the proposal when I
> >>> can, sometime in the next week.
> >>>
> >>> I agree that it would likely be opening a can of worms to create a
> >>> semantic mapping between a generalized type grammar and Arrow's
> >>> specific logical types defined in Schema.fbs. If we go down this
> >>> route, we should probably utilize the simplest possible grammar that
> >>> is capable of encoding the Type Flatbuffers union values.
> >>>
> >>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
> >>> wrote:
> >>>>
> >>>>
> >>>> I've posted a draft specification PR here, this should help orient the
> >>>> discussion a bit:
> >>>> https://github.com/apache/arrow/pull/5442
> >>>>
> >>>> Regards
> >>>>
> >>>> Antoine.
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 18 Sep 2019 19:52:38 +0200
> >>>> Antoine Pitrou <an...@python.org> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> One thing that was discussed in the sync call is the ability to easily
> >>>>> pass arrays at runtime between Arrow implementations or
> >>> Arrow-supporting
> >>>>> libraries in the same process, without bearing the cost of linking to
> >>>>> e.g. the C++ Arrow library.
> >>>>>
> >>>>> (for example: "Duckdb wants to provide an option to return Arrow data
> >>> of
> >>>>> result sets, but they don't like having Arrow as a dependency")
> >>>>>
> >>>>> One possibility would be to define a C-level protocol similar in
> >>> spirit
> >>>>> to the Python buffer protocol, which some people may be familiar with
> >>> (*).
> >>>>>
> >>>>> The basic idea is to define a simple C struct, which is ABI-stable and
> >>>>> describes an Arrow away adequately.  The struct can be
> >>> stack-allocated.
> >>>>> Its definition can also be copied in another project (or interfaced
> >>> with
> >>>>> using a C FFI layer, depending on the language).
> >>>>>
> >>>>> There is no formal proposal, this message is meant to stir the
> >>> discussion.
> >>>>>
> >>>>> Issues to work out:
> >>>>>
> >>>>> * Memory lifetime issues: where Python simply associates the Py_buffer
> >>>>> with a PyObject owner (a garbage-collected Python object), we need
> >>>>> another means to control lifetime of pointed areas.  One simple
> >>>>> possibility is to include a destructor function pointer in the
> >>> protocol
> >>>>> struct.
> >>>>>
> >>>>> * Arrow type representation.  We probably need some kind of "format"
> >>>>> mini-language to represent Arrow types, so that a type can be
> >>> described
> >>>>> using a `const char*`.  Ideally, primitives types at least should be
> >>>>> trivially parsable.  We may take inspiration from Python here
> >>> (`struct`
> >>>>> module format characters, PEP 3118 format additions).
> >>>>>
> >>>>> Example C struct definition (not a formal proposal!):
> >>>>>
> >>>>> struct ArrowBuffer {
> >>>>>   void* data;
> >>>>>   int64_t nbytes;
> >>>>>   // Called by the consumer when it doesn't need the buffer anymore
> >>>>>   void (*release)(struct ArrowBuffer*);
> >>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>   void* user_data;
> >>>>> };
> >>>>>
> >>>>> struct ArrowArray {
> >>>>>   // Type description
> >>>>>   const char* format;
> >>>>>   // Data description
> >>>>>   int64_t length;
> >>>>>   int64_t null_count;
> >>>>>   int64_t n_buffers;
> >>>>>   // Note: this pointers are probably owned by the ArrowArray struct
> >>>>>   // and will be released and free()ed by the release callback.
> >>>>>   struct BufferDescriptor* buffers;
> >>>>>   struct ArrowDescriptor* dictionary;
> >>>>>   // Called by the consumer when it doesn't need the array anymore
> >>>>>   void (*release)(struct ArrowArrayDescriptor*);
> >>>>>   // Opaque user data (for e.g. the release callback)
> >>>>>   void* user_data;
> >>>>> };
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> (*) For the record, the reference for the Python buffer protocol:
> >>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> >>>>> and its C struct definition:
> >>>>>
> >>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >>>>>
> >>>>> Regards
> >>>>>
> >>>>> Antoine.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 29/09/2019 à 06:10, Jacques Nadeau a écrit :
> * No dependency on Flatbuffers.
> * No buffer reassembly (data is already exposed in logical Arrow format).
> * Zero-copy by design.
> * Easy to reimplement from scratch.
> 
> I don't see how the flatbuffer pattern for data headers doesn't accomplish
> all of these things. At its definition, is a very simple representation of
> data that could be worked with independently of the flatbuffers codebase.
> It was designed so systems could map directly into that memory without
> interacting with a flatbuffers library.
> 
> Specifically the following three structures were designed to already allow
> what I think this proposal is trying to recreate. All three are very simple
> to construct in a direct, non-flatbuffer dependent read/write pattern.

Are they?  Personally, I wouldn't know how to do that.  I don't know
which encoding Flatbuffers use, whether it's C ABI-compatible (how could
it be? if it's portable accross different platforms, then it's probably
not compatible with any particular platform's C ABI, or only as a
conincidence), how I'm supposed to make use of the "offset" field, or
what the lifetime / ownership of all this data is.

I may be missing something, but if the answer is that it's easy to
reimplement Flatbuffers' encoding without relying on the Flatbuffers
project's source code, I'm a bit skeptical.

Regards

Antoine.


> 
> struct FieldNode {
>   length: long;
>   null_count: long;
> }
> 
> struct Buffer {
>   offset: long;
>   length: long;
> }
> 
> table RecordBatch {
>   length: long;
>   nodes: [FieldNode];
>   buffers: [Buffer];
> }
> 
> On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org> wrote:
> 
>> I'm not clear on why we need to introduce something beyond what
>> flatbuffers already provides. Can someone explain that to me? I'm not
>> really a fan of introducing a second representation of the same data (as I
>> understand it).
>>
>> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com> wrote:
>>
>>> This is helpful, I will leave some comments on the proposal when I
>>> can, sometime in the next week.
>>>
>>> I agree that it would likely be opening a can of worms to create a
>>> semantic mapping between a generalized type grammar and Arrow's
>>> specific logical types defined in Schema.fbs. If we go down this
>>> route, we should probably utilize the simplest possible grammar that
>>> is capable of encoding the Type Flatbuffers union values.
>>>
>>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
>>> wrote:
>>>>
>>>>
>>>> I've posted a draft specification PR here, this should help orient the
>>>> discussion a bit:
>>>> https://github.com/apache/arrow/pull/5442
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>>
>>>> On Wed, 18 Sep 2019 19:52:38 +0200
>>>> Antoine Pitrou <an...@python.org> wrote:
>>>>> Hello,
>>>>>
>>>>> One thing that was discussed in the sync call is the ability to easily
>>>>> pass arrays at runtime between Arrow implementations or
>>> Arrow-supporting
>>>>> libraries in the same process, without bearing the cost of linking to
>>>>> e.g. the C++ Arrow library.
>>>>>
>>>>> (for example: "Duckdb wants to provide an option to return Arrow data
>>> of
>>>>> result sets, but they don't like having Arrow as a dependency")
>>>>>
>>>>> One possibility would be to define a C-level protocol similar in
>>> spirit
>>>>> to the Python buffer protocol, which some people may be familiar with
>>> (*).
>>>>>
>>>>> The basic idea is to define a simple C struct, which is ABI-stable and
>>>>> describes an Arrow away adequately.  The struct can be
>>> stack-allocated.
>>>>> Its definition can also be copied in another project (or interfaced
>>> with
>>>>> using a C FFI layer, depending on the language).
>>>>>
>>>>> There is no formal proposal, this message is meant to stir the
>>> discussion.
>>>>>
>>>>> Issues to work out:
>>>>>
>>>>> * Memory lifetime issues: where Python simply associates the Py_buffer
>>>>> with a PyObject owner (a garbage-collected Python object), we need
>>>>> another means to control lifetime of pointed areas.  One simple
>>>>> possibility is to include a destructor function pointer in the
>>> protocol
>>>>> struct.
>>>>>
>>>>> * Arrow type representation.  We probably need some kind of "format"
>>>>> mini-language to represent Arrow types, so that a type can be
>>> described
>>>>> using a `const char*`.  Ideally, primitives types at least should be
>>>>> trivially parsable.  We may take inspiration from Python here
>>> (`struct`
>>>>> module format characters, PEP 3118 format additions).
>>>>>
>>>>> Example C struct definition (not a formal proposal!):
>>>>>
>>>>> struct ArrowBuffer {
>>>>>   void* data;
>>>>>   int64_t nbytes;
>>>>>   // Called by the consumer when it doesn't need the buffer anymore
>>>>>   void (*release)(struct ArrowBuffer*);
>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>   void* user_data;
>>>>> };
>>>>>
>>>>> struct ArrowArray {
>>>>>   // Type description
>>>>>   const char* format;
>>>>>   // Data description
>>>>>   int64_t length;
>>>>>   int64_t null_count;
>>>>>   int64_t n_buffers;
>>>>>   // Note: this pointers are probably owned by the ArrowArray struct
>>>>>   // and will be released and free()ed by the release callback.
>>>>>   struct BufferDescriptor* buffers;
>>>>>   struct ArrowDescriptor* dictionary;
>>>>>   // Called by the consumer when it doesn't need the array anymore
>>>>>   void (*release)(struct ArrowArrayDescriptor*);
>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>   void* user_data;
>>>>> };
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> (*) For the record, the reference for the Python buffer protocol:
>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>>>>> and its C struct definition:
>>>>>
>>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>
>>>>
>>>>
>>>
>>
> 

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

* No dependency on Flatbuffers.
* No buffer reassembly (data is already exposed in logical Arrow format).
* Zero-copy by design.
* Easy to reimplement from scratch.

I don't see how the flatbuffer pattern for data headers doesn't accomplish
all of these things. At its definition, is a very simple representation of
data that could be worked with independently of the flatbuffers codebase.
It was designed so systems could map directly into that memory without
interacting with a flatbuffers library.

Specifically the following three structures were designed to already allow
what I think this proposal is trying to recreate. All three are very simple
to construct in a direct, non-flatbuffer dependent read/write pattern.

struct FieldNode {
  length: long;
  null_count: long;
}

struct Buffer {
  offset: long;
  length: long;
}

table RecordBatch {
  length: long;
  nodes: [FieldNode];
  buffers: [Buffer];
}

On Sat, Sep 28, 2019 at 9:02 PM Jacques Nadeau <ja...@apache.org> wrote:

> I'm not clear on why we need to introduce something beyond what
> flatbuffers already provides. Can someone explain that to me? I'm not
> really a fan of introducing a second representation of the same data (as I
> understand it).
>
> On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com> wrote:
>
>> This is helpful, I will leave some comments on the proposal when I
>> can, sometime in the next week.
>>
>> I agree that it would likely be opening a can of worms to create a
>> semantic mapping between a generalized type grammar and Arrow's
>> specific logical types defined in Schema.fbs. If we go down this
>> route, we should probably utilize the simplest possible grammar that
>> is capable of encoding the Type Flatbuffers union values.
>>
>> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
>> wrote:
>> >
>> >
>> > I've posted a draft specification PR here, this should help orient the
>> > discussion a bit:
>> > https://github.com/apache/arrow/pull/5442
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> >
>> > On Wed, 18 Sep 2019 19:52:38 +0200
>> > Antoine Pitrou <an...@python.org> wrote:
>> > > Hello,
>> > >
>> > > One thing that was discussed in the sync call is the ability to easily
>> > > pass arrays at runtime between Arrow implementations or
>> Arrow-supporting
>> > > libraries in the same process, without bearing the cost of linking to
>> > > e.g. the C++ Arrow library.
>> > >
>> > > (for example: "Duckdb wants to provide an option to return Arrow data
>> of
>> > > result sets, but they don't like having Arrow as a dependency")
>> > >
>> > > One possibility would be to define a C-level protocol similar in
>> spirit
>> > > to the Python buffer protocol, which some people may be familiar with
>> (*).
>> > >
>> > > The basic idea is to define a simple C struct, which is ABI-stable and
>> > > describes an Arrow away adequately.  The struct can be
>> stack-allocated.
>> > > Its definition can also be copied in another project (or interfaced
>> with
>> > > using a C FFI layer, depending on the language).
>> > >
>> > > There is no formal proposal, this message is meant to stir the
>> discussion.
>> > >
>> > > Issues to work out:
>> > >
>> > > * Memory lifetime issues: where Python simply associates the Py_buffer
>> > > with a PyObject owner (a garbage-collected Python object), we need
>> > > another means to control lifetime of pointed areas.  One simple
>> > > possibility is to include a destructor function pointer in the
>> protocol
>> > > struct.
>> > >
>> > > * Arrow type representation.  We probably need some kind of "format"
>> > > mini-language to represent Arrow types, so that a type can be
>> described
>> > > using a `const char*`.  Ideally, primitives types at least should be
>> > > trivially parsable.  We may take inspiration from Python here
>> (`struct`
>> > > module format characters, PEP 3118 format additions).
>> > >
>> > > Example C struct definition (not a formal proposal!):
>> > >
>> > > struct ArrowBuffer {
>> > >   void* data;
>> > >   int64_t nbytes;
>> > >   // Called by the consumer when it doesn't need the buffer anymore
>> > >   void (*release)(struct ArrowBuffer*);
>> > >   // Opaque user data (for e.g. the release callback)
>> > >   void* user_data;
>> > > };
>> > >
>> > > struct ArrowArray {
>> > >   // Type description
>> > >   const char* format;
>> > >   // Data description
>> > >   int64_t length;
>> > >   int64_t null_count;
>> > >   int64_t n_buffers;
>> > >   // Note: this pointers are probably owned by the ArrowArray struct
>> > >   // and will be released and free()ed by the release callback.
>> > >   struct BufferDescriptor* buffers;
>> > >   struct ArrowDescriptor* dictionary;
>> > >   // Called by the consumer when it doesn't need the array anymore
>> > >   void (*release)(struct ArrowArrayDescriptor*);
>> > >   // Opaque user data (for e.g. the release callback)
>> > >   void* user_data;
>> > > };
>> > >
>> > > Thoughts?
>> > >
>> > > (*) For the record, the reference for the Python buffer protocol:
>> > > https://docs.python.org/3/c-api/buffer.html#buffer-structure
>> > > and its C struct definition:
>> > >
>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> >
>> >
>> >
>>
>

Re: [DISCUSS] C-level in-process array protocol

[Jacques Nadeau <ja...@apache.org>]

I'm not clear on why we need to introduce something beyond what flatbuffers
already provides. Can someone explain that to me? I'm not really a fan of
introducing a second representation of the same data (as I understand it).

On Thu, Sep 19, 2019 at 1:15 PM Wes McKinney <we...@gmail.com> wrote:

> This is helpful, I will leave some comments on the proposal when I
> can, sometime in the next week.
>
> I agree that it would likely be opening a can of worms to create a
> semantic mapping between a generalized type grammar and Arrow's
> specific logical types defined in Schema.fbs. If we go down this
> route, we should probably utilize the simplest possible grammar that
> is capable of encoding the Type Flatbuffers union values.
>
> On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net>
> wrote:
> >
> >
> > I've posted a draft specification PR here, this should help orient the
> > discussion a bit:
> > https://github.com/apache/arrow/pull/5442
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > On Wed, 18 Sep 2019 19:52:38 +0200
> > Antoine Pitrou <an...@python.org> wrote:
> > > Hello,
> > >
> > > One thing that was discussed in the sync call is the ability to easily
> > > pass arrays at runtime between Arrow implementations or
> Arrow-supporting
> > > libraries in the same process, without bearing the cost of linking to
> > > e.g. the C++ Arrow library.
> > >
> > > (for example: "Duckdb wants to provide an option to return Arrow data
> of
> > > result sets, but they don't like having Arrow as a dependency")
> > >
> > > One possibility would be to define a C-level protocol similar in spirit
> > > to the Python buffer protocol, which some people may be familiar with
> (*).
> > >
> > > The basic idea is to define a simple C struct, which is ABI-stable and
> > > describes an Arrow away adequately.  The struct can be stack-allocated.
> > > Its definition can also be copied in another project (or interfaced
> with
> > > using a C FFI layer, depending on the language).
> > >
> > > There is no formal proposal, this message is meant to stir the
> discussion.
> > >
> > > Issues to work out:
> > >
> > > * Memory lifetime issues: where Python simply associates the Py_buffer
> > > with a PyObject owner (a garbage-collected Python object), we need
> > > another means to control lifetime of pointed areas.  One simple
> > > possibility is to include a destructor function pointer in the protocol
> > > struct.
> > >
> > > * Arrow type representation.  We probably need some kind of "format"
> > > mini-language to represent Arrow types, so that a type can be described
> > > using a `const char*`.  Ideally, primitives types at least should be
> > > trivially parsable.  We may take inspiration from Python here (`struct`
> > > module format characters, PEP 3118 format additions).
> > >
> > > Example C struct definition (not a formal proposal!):
> > >
> > > struct ArrowBuffer {
> > >   void* data;
> > >   int64_t nbytes;
> > >   // Called by the consumer when it doesn't need the buffer anymore
> > >   void (*release)(struct ArrowBuffer*);
> > >   // Opaque user data (for e.g. the release callback)
> > >   void* user_data;
> > > };
> > >
> > > struct ArrowArray {
> > >   // Type description
> > >   const char* format;
> > >   // Data description
> > >   int64_t length;
> > >   int64_t null_count;
> > >   int64_t n_buffers;
> > >   // Note: this pointers are probably owned by the ArrowArray struct
> > >   // and will be released and free()ed by the release callback.
> > >   struct BufferDescriptor* buffers;
> > >   struct ArrowDescriptor* dictionary;
> > >   // Called by the consumer when it doesn't need the array anymore
> > >   void (*release)(struct ArrowArrayDescriptor*);
> > >   // Opaque user data (for e.g. the release callback)
> > >   void* user_data;
> > > };
> > >
> > > Thoughts?
> > >
> > > (*) For the record, the reference for the Python buffer protocol:
> > > https://docs.python.org/3/c-api/buffer.html#buffer-structure
> > > and its C struct definition:
> > >
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> >
> >
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Wes McKinney <we...@gmail.com>]

This is helpful, I will leave some comments on the proposal when I
can, sometime in the next week.

I agree that it would likely be opening a can of worms to create a
semantic mapping between a generalized type grammar and Arrow's
specific logical types defined in Schema.fbs. If we go down this
route, we should probably utilize the simplest possible grammar that
is capable of encoding the Type Flatbuffers union values.

On Thu, Sep 19, 2019 at 2:49 PM Antoine Pitrou <so...@pitrou.net> wrote:
>
>
> I've posted a draft specification PR here, this should help orient the
> discussion a bit:
> https://github.com/apache/arrow/pull/5442
>
> Regards
>
> Antoine.
>
>
>
> On Wed, 18 Sep 2019 19:52:38 +0200
> Antoine Pitrou <an...@python.org> wrote:
> > Hello,
> >
> > One thing that was discussed in the sync call is the ability to easily
> > pass arrays at runtime between Arrow implementations or Arrow-supporting
> > libraries in the same process, without bearing the cost of linking to
> > e.g. the C++ Arrow library.
> >
> > (for example: "Duckdb wants to provide an option to return Arrow data of
> > result sets, but they don't like having Arrow as a dependency")
> >
> > One possibility would be to define a C-level protocol similar in spirit
> > to the Python buffer protocol, which some people may be familiar with (*).
> >
> > The basic idea is to define a simple C struct, which is ABI-stable and
> > describes an Arrow away adequately.  The struct can be stack-allocated.
> > Its definition can also be copied in another project (or interfaced with
> > using a C FFI layer, depending on the language).
> >
> > There is no formal proposal, this message is meant to stir the discussion.
> >
> > Issues to work out:
> >
> > * Memory lifetime issues: where Python simply associates the Py_buffer
> > with a PyObject owner (a garbage-collected Python object), we need
> > another means to control lifetime of pointed areas.  One simple
> > possibility is to include a destructor function pointer in the protocol
> > struct.
> >
> > * Arrow type representation.  We probably need some kind of "format"
> > mini-language to represent Arrow types, so that a type can be described
> > using a `const char*`.  Ideally, primitives types at least should be
> > trivially parsable.  We may take inspiration from Python here (`struct`
> > module format characters, PEP 3118 format additions).
> >
> > Example C struct definition (not a formal proposal!):
> >
> > struct ArrowBuffer {
> >   void* data;
> >   int64_t nbytes;
> >   // Called by the consumer when it doesn't need the buffer anymore
> >   void (*release)(struct ArrowBuffer*);
> >   // Opaque user data (for e.g. the release callback)
> >   void* user_data;
> > };
> >
> > struct ArrowArray {
> >   // Type description
> >   const char* format;
> >   // Data description
> >   int64_t length;
> >   int64_t null_count;
> >   int64_t n_buffers;
> >   // Note: this pointers are probably owned by the ArrowArray struct
> >   // and will be released and free()ed by the release callback.
> >   struct BufferDescriptor* buffers;
> >   struct ArrowDescriptor* dictionary;
> >   // Called by the consumer when it doesn't need the array anymore
> >   void (*release)(struct ArrowArrayDescriptor*);
> >   // Opaque user data (for e.g. the release callback)
> >   void* user_data;
> > };
> >
> > Thoughts?
> >
> > (*) For the record, the reference for the Python buffer protocol:
> > https://docs.python.org/3/c-api/buffer.html#buffer-structure
> > and its C struct definition:
> > https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >
> > Regards
> >
> > Antoine.
> >
>
>
>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <so...@pitrou.net>]

I've posted a draft specification PR here, this should help orient the
discussion a bit:
https://github.com/apache/arrow/pull/5442

Regards

Antoine.



On Wed, 18 Sep 2019 19:52:38 +0200
Antoine Pitrou <an...@python.org> wrote:
> Hello,
> 
> One thing that was discussed in the sync call is the ability to easily
> pass arrays at runtime between Arrow implementations or Arrow-supporting
> libraries in the same process, without bearing the cost of linking to
> e.g. the C++ Arrow library.
> 
> (for example: "Duckdb wants to provide an option to return Arrow data of
> result sets, but they don't like having Arrow as a dependency")
> 
> One possibility would be to define a C-level protocol similar in spirit
> to the Python buffer protocol, which some people may be familiar with (*).
> 
> The basic idea is to define a simple C struct, which is ABI-stable and
> describes an Arrow away adequately.  The struct can be stack-allocated.
> Its definition can also be copied in another project (or interfaced with
> using a C FFI layer, depending on the language).
> 
> There is no formal proposal, this message is meant to stir the discussion.
> 
> Issues to work out:
> 
> * Memory lifetime issues: where Python simply associates the Py_buffer
> with a PyObject owner (a garbage-collected Python object), we need
> another means to control lifetime of pointed areas.  One simple
> possibility is to include a destructor function pointer in the protocol
> struct.
> 
> * Arrow type representation.  We probably need some kind of "format"
> mini-language to represent Arrow types, so that a type can be described
> using a `const char*`.  Ideally, primitives types at least should be
> trivially parsable.  We may take inspiration from Python here (`struct`
> module format characters, PEP 3118 format additions).
> 
> Example C struct definition (not a formal proposal!):
> 
> struct ArrowBuffer {
>   void* data;
>   int64_t nbytes;
>   // Called by the consumer when it doesn't need the buffer anymore
>   void (*release)(struct ArrowBuffer*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
> 
> struct ArrowArray {
>   // Type description
>   const char* format;
>   // Data description
>   int64_t length;
>   int64_t null_count;
>   int64_t n_buffers;
>   // Note: this pointers are probably owned by the ArrowArray struct
>   // and will be released and free()ed by the release callback.
>   struct BufferDescriptor* buffers;
>   struct ArrowDescriptor* dictionary;
>   // Called by the consumer when it doesn't need the array anymore
>   void (*release)(struct ArrowArrayDescriptor*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
> 
> Thoughts?
> 
> (*) For the record, the reference for the Python buffer protocol:
> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> and its C struct definition:
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> 
> Regards
> 
> Antoine.
> 

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

I suppose it could be possible for an Arrow array to describe itself
using the ndtypes vocabulary at some point.  However, this is
non-trivial, both on the producer and consumer side.  Moreover, both
sides must ensure they use the same ndtypes description.

The idea here is to have a C data protocol, without any need for a
helper C library, that's a simple as possible and directly expresses the
Arrow data without needing any semantic mapping.  Also it should allow
transmission via FFI layers with as little complication as possible.

Which is why it most probably needs to be Arrow-specific.

Regards

Antoine.


Le 19/09/2019 à 21:14, Travis Oliphant a écrit :
> I know some on this list are familiar, but many may not have seen ndtypes
> in xnd:  https://github.com/xnd-project/ndtypes
> 
> It generalizes PEP 3118 for cross-language data-structure handling.
> 
> Either a dependency on the small C-library libndtypes or using the concepts
> could be done.
> 
> -Travis
> 
> 
> On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Hello,
>>
>> One thing that was discussed in the sync call is the ability to easily
>> pass arrays at runtime between Arrow implementations or Arrow-supporting
>> libraries in the same process, without bearing the cost of linking to
>> e.g. the C++ Arrow library.
>>
>> (for example: "Duckdb wants to provide an option to return Arrow data of
>> result sets, but they don't like having Arrow as a dependency")
>>
>> One possibility would be to define a C-level protocol similar in spirit
>> to the Python buffer protocol, which some people may be familiar with (*).
>>
>> The basic idea is to define a simple C struct, which is ABI-stable and
>> describes an Arrow away adequately.  The struct can be stack-allocated.
>> Its definition can also be copied in another project (or interfaced with
>> using a C FFI layer, depending on the language).
>>
>> There is no formal proposal, this message is meant to stir the discussion.
>>
>> Issues to work out:
>>
>> * Memory lifetime issues: where Python simply associates the Py_buffer
>> with a PyObject owner (a garbage-collected Python object), we need
>> another means to control lifetime of pointed areas.  One simple
>> possibility is to include a destructor function pointer in the protocol
>> struct.
>>
>> * Arrow type representation.  We probably need some kind of "format"
>> mini-language to represent Arrow types, so that a type can be described
>> using a `const char*`.  Ideally, primitives types at least should be
>> trivially parsable.  We may take inspiration from Python here (`struct`
>> module format characters, PEP 3118 format additions).
>>
>> Example C struct definition (not a formal proposal!):
>>
>> struct ArrowBuffer {
>>   void* data;
>>   int64_t nbytes;
>>   // Called by the consumer when it doesn't need the buffer anymore
>>   void (*release)(struct ArrowBuffer*);
>>   // Opaque user data (for e.g. the release callback)
>>   void* user_data;
>> };
>>
>> struct ArrowArray {
>>   // Type description
>>   const char* format;
>>   // Data description
>>   int64_t length;
>>   int64_t null_count;
>>   int64_t n_buffers;
>>   // Note: this pointers are probably owned by the ArrowArray struct
>>   // and will be released and free()ed by the release callback.
>>   struct BufferDescriptor* buffers;
>>   struct ArrowDescriptor* dictionary;
>>   // Called by the consumer when it doesn't need the array anymore
>>   void (*release)(struct ArrowArrayDescriptor*);
>>   // Opaque user data (for e.g. the release callback)
>>   void* user_data;
>> };
>>
>> Thoughts?
>>
>> (*) For the record, the reference for the Python buffer protocol:
>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>> and its C struct definition:
>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>
>> Regards
>>
>> Antoine.
>>
> 
> 

Re: [DISCUSS] C-level in-process array protocol

[Travis Oliphant <tr...@quansight.com>]

I know some on this list are familiar, but many may not have seen ndtypes
in xnd:  https://github.com/xnd-project/ndtypes

It generalizes PEP 3118 for cross-language data-structure handling.

Either a dependency on the small C-library libndtypes or using the concepts
could be done.

-Travis


On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello,
>
> One thing that was discussed in the sync call is the ability to easily
> pass arrays at runtime between Arrow implementations or Arrow-supporting
> libraries in the same process, without bearing the cost of linking to
> e.g. the C++ Arrow library.
>
> (for example: "Duckdb wants to provide an option to return Arrow data of
> result sets, but they don't like having Arrow as a dependency")
>
> One possibility would be to define a C-level protocol similar in spirit
> to the Python buffer protocol, which some people may be familiar with (*).
>
> The basic idea is to define a simple C struct, which is ABI-stable and
> describes an Arrow away adequately.  The struct can be stack-allocated.
> Its definition can also be copied in another project (or interfaced with
> using a C FFI layer, depending on the language).
>
> There is no formal proposal, this message is meant to stir the discussion.
>
> Issues to work out:
>
> * Memory lifetime issues: where Python simply associates the Py_buffer
> with a PyObject owner (a garbage-collected Python object), we need
> another means to control lifetime of pointed areas.  One simple
> possibility is to include a destructor function pointer in the protocol
> struct.
>
> * Arrow type representation.  We probably need some kind of "format"
> mini-language to represent Arrow types, so that a type can be described
> using a `const char*`.  Ideally, primitives types at least should be
> trivially parsable.  We may take inspiration from Python here (`struct`
> module format characters, PEP 3118 format additions).
>
> Example C struct definition (not a formal proposal!):
>
> struct ArrowBuffer {
>   void* data;
>   int64_t nbytes;
>   // Called by the consumer when it doesn't need the buffer anymore
>   void (*release)(struct ArrowBuffer*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
>
> struct ArrowArray {
>   // Type description
>   const char* format;
>   // Data description
>   int64_t length;
>   int64_t null_count;
>   int64_t n_buffers;
>   // Note: this pointers are probably owned by the ArrowArray struct
>   // and will be released and free()ed by the release callback.
>   struct BufferDescriptor* buffers;
>   struct ArrowDescriptor* dictionary;
>   // Called by the consumer when it doesn't need the array anymore
>   void (*release)(struct ArrowArrayDescriptor*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
>
> Thoughts?
>
> (*) For the record, the reference for the Python buffer protocol:
> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> and its C struct definition:
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>
> Regards
>
> Antoine.
>


-- 

*Travis Oliphant*
CEO
512 826 7480

<https://www.quansight.com/>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 19/09/2019 à 19:11, Uwe L. Korn a écrit :
> Hello,
> 
> I like this proposal as it will make interfacing inside a process between various Arrow supports much easier. I'm a bit critical though of using a string as the format representation as one needs to parse it correctly. Couldn't we use the enums we already have and reimplement them as C-defines instead?

We could, but then we need to represent type parameters separately, as
some types are parametric (such as Time-related types).  So we would
still have some kind of encoded representation for those parameters.

So it may be as easy to represent everything inside the format string:
the type class (a single character perhaps) and optionally the type
instance parameters (if necessary).

Note that for non-parametric primitive types such as int64_t, double,
utf8... the format string will be a single character anyway.

Regards

Antoine.


> 
> Uwe
> 
> On Thu, Sep 19, 2019, at 6:21 PM, Zhuo Peng wrote:
>> Hi Antoine,
>>
>> I'm also interested in a stable ABI (previously I posted on this mailing
>> list about the ABI issues I had [1]). Does having such an ABI-stable
>> C-struct imply that there will be a set of C-APIs exposed by the Arrow
>> (C++) library (which I think would lead to a solution to all the inherit
>> ABI issues caused by C++)?
>>
>> [1]
>> https://lists.apache.org/thread.html/27b6e2a30cf93c5f5f78de970c68c7d996f538d94ab61431fa342f41@%3Cdev.arrow.apache.org%3E
>>
>> On Thu, Sep 19, 2019 at 1:07 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>>
>>> Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
>>>> I like the idea of a stable ABI for in-processing  that can be used for
>>> in
>>>> process communication.  For instance, there was a recent question on
>>>> stack-overflow on how to solve this [1].
>>>>
>>>> A couple of thoughts/questions:
>>>> * Would ArrowArray also need a self reference for children arrays?
>>>
>>> Yes, I forgot that.  I also think we don't need a separate Buffer
>>> struct, instead the Array struct should own all its buffers.
>>>
>>>> * Should transferring key-value metadata be in scope?
>>>
>>> Yes.  It could either be in the format string or a separate string.  The
>>> upside of a separate string is that a consumer may ignore it trivially
>>> if it doesn't need the information.
>>>
>>> Another open question is for nested types: does the format string
>>> represent the entire type including children?  Or must child types be
>>> read in the child arrays?  If we mimick ArrayData, then the format
>>> string should represent the entire type; it will then be more complex to
>>> parse.
>>>
>>> We should also make sure that extension types fit in the protocol.
>>>
>>>> * Should the API more closely align the IPC spec (pass a schema
>>> separately
>>>> and list of buffers instead of individual arrays)?
>>>
>>> Then you have that's not immediately usable (you have to do some
>>> processing to reconstitute the individual arrays).  One goal here is to
>>> minimize implementation costs for producers and consumers.  The
>>> assumption is a data model similar to the C++ ArrowData model; do we
>>> have implementations that use an entirely different model?  Perhaps I
>>> should take a look :-)
>>>
>>> Note that the draft I posted only concerns arrays.  We may also want to
>>> have a C struct for batches or tables.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>>>
>>>> Thanks,
>>>> Micah
>>>>
>>>> [1]
>>>>
>>> https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
>>>>
>>>> On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org>
>>> wrote:
>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> One thing that was discussed in the sync call is the ability to easily
>>>>> pass arrays at runtime between Arrow implementations or Arrow-supporting
>>>>> libraries in the same process, without bearing the cost of linking to
>>>>> e.g. the C++ Arrow library.
>>>>>
>>>>> (for example: "Duckdb wants to provide an option to return Arrow data of
>>>>> result sets, but they don't like having Arrow as a dependency")
>>>>>
>>>>> One possibility would be to define a C-level protocol similar in spirit
>>>>> to the Python buffer protocol, which some people may be familiar with
>>> (*).
>>>>>
>>>>> The basic idea is to define a simple C struct, which is ABI-stable and
>>>>> describes an Arrow away adequately.  The struct can be stack-allocated.
>>>>> Its definition can also be copied in another project (or interfaced with
>>>>> using a C FFI layer, depending on the language).
>>>>>
>>>>> There is no formal proposal, this message is meant to stir the
>>> discussion.
>>>>>
>>>>> Issues to work out:
>>>>>
>>>>> * Memory lifetime issues: where Python simply associates the Py_buffer
>>>>> with a PyObject owner (a garbage-collected Python object), we need
>>>>> another means to control lifetime of pointed areas.  One simple
>>>>> possibility is to include a destructor function pointer in the protocol
>>>>> struct.
>>>>>
>>>>> * Arrow type representation.  We probably need some kind of "format"
>>>>> mini-language to represent Arrow types, so that a type can be described
>>>>> using a `const char*`.  Ideally, primitives types at least should be
>>>>> trivially parsable.  We may take inspiration from Python here (`struct`
>>>>> module format characters, PEP 3118 format additions).
>>>>>
>>>>> Example C struct definition (not a formal proposal!):
>>>>>
>>>>> struct ArrowBuffer {
>>>>>   void* data;
>>>>>   int64_t nbytes;
>>>>>   // Called by the consumer when it doesn't need the buffer anymore
>>>>>   void (*release)(struct ArrowBuffer*);
>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>   void* user_data;
>>>>> };
>>>>>
>>>>> struct ArrowArray {
>>>>>   // Type description
>>>>>   const char* format;
>>>>>   // Data description
>>>>>   int64_t length;
>>>>>   int64_t null_count;
>>>>>   int64_t n_buffers;
>>>>>   // Note: this pointers are probably owned by the ArrowArray struct
>>>>>   // and will be released and free()ed by the release callback.
>>>>>   struct BufferDescriptor* buffers;
>>>>>   struct ArrowDescriptor* dictionary;
>>>>>   // Called by the consumer when it doesn't need the array anymore
>>>>>   void (*release)(struct ArrowArrayDescriptor*);
>>>>>   // Opaque user data (for e.g. the release callback)
>>>>>   void* user_data;
>>>>> };
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> (*) For the record, the reference for the Python buffer protocol:
>>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>>>>> and its C struct definition:
>>>>>
>>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>
>>>
>>

Re: [DISCUSS] C-level in-process array protocol

["Uwe L. Korn" <uw...@xhochy.com>]

Hello,

I like this proposal as it will make interfacing inside a process between various Arrow supports much easier. I'm a bit critical though of using a string as the format representation as one needs to parse it correctly. Couldn't we use the enums we already have and reimplement them as C-defines instead?

Uwe

On Thu, Sep 19, 2019, at 6:21 PM, Zhuo Peng wrote:
> Hi Antoine,
> 
> I'm also interested in a stable ABI (previously I posted on this mailing
> list about the ABI issues I had [1]). Does having such an ABI-stable
> C-struct imply that there will be a set of C-APIs exposed by the Arrow
> (C++) library (which I think would lead to a solution to all the inherit
> ABI issues caused by C++)?
> 
> [1]
> https://lists.apache.org/thread.html/27b6e2a30cf93c5f5f78de970c68c7d996f538d94ab61431fa342f41@%3Cdev.arrow.apache.org%3E
> 
> On Thu, Sep 19, 2019 at 1:07 AM Antoine Pitrou <an...@python.org> wrote:
> 
> >
> > Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
> > > I like the idea of a stable ABI for in-processing  that can be used for
> > in
> > > process communication.  For instance, there was a recent question on
> > > stack-overflow on how to solve this [1].
> > >
> > > A couple of thoughts/questions:
> > > * Would ArrowArray also need a self reference for children arrays?
> >
> > Yes, I forgot that.  I also think we don't need a separate Buffer
> > struct, instead the Array struct should own all its buffers.
> >
> > > * Should transferring key-value metadata be in scope?
> >
> > Yes.  It could either be in the format string or a separate string.  The
> > upside of a separate string is that a consumer may ignore it trivially
> > if it doesn't need the information.
> >
> > Another open question is for nested types: does the format string
> > represent the entire type including children?  Or must child types be
> > read in the child arrays?  If we mimick ArrayData, then the format
> > string should represent the entire type; it will then be more complex to
> > parse.
> >
> > We should also make sure that extension types fit in the protocol.
> >
> > > * Should the API more closely align the IPC spec (pass a schema
> > separately
> > > and list of buffers instead of individual arrays)?
> >
> > Then you have that's not immediately usable (you have to do some
> > processing to reconstitute the individual arrays).  One goal here is to
> > minimize implementation costs for producers and consumers.  The
> > assumption is a data model similar to the C++ ArrowData model; do we
> > have implementations that use an entirely different model?  Perhaps I
> > should take a look :-)
> >
> > Note that the draft I posted only concerns arrays.  We may also want to
> > have a C struct for batches or tables.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > >
> > > Thanks,
> > > Micah
> > >
> > > [1]
> > >
> > https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
> > >
> > > On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> Hello,
> > >>
> > >> One thing that was discussed in the sync call is the ability to easily
> > >> pass arrays at runtime between Arrow implementations or Arrow-supporting
> > >> libraries in the same process, without bearing the cost of linking to
> > >> e.g. the C++ Arrow library.
> > >>
> > >> (for example: "Duckdb wants to provide an option to return Arrow data of
> > >> result sets, but they don't like having Arrow as a dependency")
> > >>
> > >> One possibility would be to define a C-level protocol similar in spirit
> > >> to the Python buffer protocol, which some people may be familiar with
> > (*).
> > >>
> > >> The basic idea is to define a simple C struct, which is ABI-stable and
> > >> describes an Arrow away adequately.  The struct can be stack-allocated.
> > >> Its definition can also be copied in another project (or interfaced with
> > >> using a C FFI layer, depending on the language).
> > >>
> > >> There is no formal proposal, this message is meant to stir the
> > discussion.
> > >>
> > >> Issues to work out:
> > >>
> > >> * Memory lifetime issues: where Python simply associates the Py_buffer
> > >> with a PyObject owner (a garbage-collected Python object), we need
> > >> another means to control lifetime of pointed areas.  One simple
> > >> possibility is to include a destructor function pointer in the protocol
> > >> struct.
> > >>
> > >> * Arrow type representation.  We probably need some kind of "format"
> > >> mini-language to represent Arrow types, so that a type can be described
> > >> using a `const char*`.  Ideally, primitives types at least should be
> > >> trivially parsable.  We may take inspiration from Python here (`struct`
> > >> module format characters, PEP 3118 format additions).
> > >>
> > >> Example C struct definition (not a formal proposal!):
> > >>
> > >> struct ArrowBuffer {
> > >>   void* data;
> > >>   int64_t nbytes;
> > >>   // Called by the consumer when it doesn't need the buffer anymore
> > >>   void (*release)(struct ArrowBuffer*);
> > >>   // Opaque user data (for e.g. the release callback)
> > >>   void* user_data;
> > >> };
> > >>
> > >> struct ArrowArray {
> > >>   // Type description
> > >>   const char* format;
> > >>   // Data description
> > >>   int64_t length;
> > >>   int64_t null_count;
> > >>   int64_t n_buffers;
> > >>   // Note: this pointers are probably owned by the ArrowArray struct
> > >>   // and will be released and free()ed by the release callback.
> > >>   struct BufferDescriptor* buffers;
> > >>   struct ArrowDescriptor* dictionary;
> > >>   // Called by the consumer when it doesn't need the array anymore
> > >>   void (*release)(struct ArrowArrayDescriptor*);
> > >>   // Opaque user data (for e.g. the release callback)
> > >>   void* user_data;
> > >> };
> > >>
> > >> Thoughts?
> > >>
> > >> (*) For the record, the reference for the Python buffer protocol:
> > >> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> > >> and its C struct definition:
> > >>
> > https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Zhuo Peng <br...@gmail.com>]

On Thu, Sep 19, 2019 at 10:56 Antoine Pitrou <an...@python.org> wrote:

>
> Le 19/09/2019 à 19:52, Zhuo Peng a écrit :
> >
> > The problems are only potential and theoretical, and won't bite anyone
> > until it occurs though, and it's more likely to happen with pip/wheel
> than
> > with conda.
> >
> > But anyways, this idea is still nice. I could imagine at least in arrow's
> > Python-C-API, there would be a
> >
> > PyObject* pyarrow_array_from_c_protocol(ArrayArray*);
> >
> > this way the C++ APIs can be avoided while still allowing arrays to be
> > created in C/C++ and used in python.
>
> Adding a Python C API function is a nice idea.
> However, I *still* don't understand how it will solve your problem.  The
> Cython modules comprising PyArrow will still call the C++ APIs, with the
> ABI problems that entails.

Those calls are internal to libarrow.so and libarrow_python.so which always
agrees on the ABI.

It’s different from the client library having to create an arrow::Array
which may contain, say a std::vector<int> from gcc5, then pass it to an
Arrow C++ API exposed by libarrow.so, whose definition of std::vector<int>
is from gcc7.

>
>
> Regards
>
> Antoine.
>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 19/09/2019 à 19:52, Zhuo Peng a écrit :
> 
> The problems are only potential and theoretical, and won't bite anyone
> until it occurs though, and it's more likely to happen with pip/wheel than
> with conda.
> 
> But anyways, this idea is still nice. I could imagine at least in arrow's
> Python-C-API, there would be a
> 
> PyObject* pyarrow_array_from_c_protocol(ArrayArray*);
> 
> this way the C++ APIs can be avoided while still allowing arrays to be
> created in C/C++ and used in python.

Adding a Python C API function is a nice idea.
However, I *still* don't understand how it will solve your problem.  The
Cython modules comprising PyArrow will still call the C++ APIs, with the
ABI problems that entails.

Regards

Antoine.

Re: [DISCUSS] C-level in-process array protocol

[Zhuo Peng <zh...@sealscript.net>]

On Thu, Sep 19, 2019 at 10:18 AM Antoine Pitrou <an...@python.org> wrote:

>
> No, the plan for this proposal is to avoid providing a C API.  Each
> Arrow implementation could produce and consume the C data protocol, for
> example the C++ Array class could add these methods:
>
> class Array {
>   // ...
>
>  public:
>   // Export array to the C data protocol
>   void Share(ArrowArray* out);
>   // Import a C data protocol array
>   static Status FromShared(ArrowArray* input,
>                            std::shared_ptr<Array>* out);
> };
>
> Also, I don't know why a C API exposed by the C++ library would solve
> your problem.  You would still have a problem with bundling the .so,
> symbol conflicts if several libraries load libarrow.so, etc.

The problem is mainly about C++ not being able to provide a stable ABI for
templates (thus STL). If Arrow C++ library's public headers contain
templates or definitions from STL, the only way to guarantee safety is to
force the client library use the same toolchain and the same flags with
which the Arrow DSO was built. (Yes, distribution methods like Conda help
mitigate that issue by enforcing a uniform toolchain (almost), but problems
can still occur, if, say a client is built with --std=c++17 while
libarrow.so is built with --std=gnu11 (example at [1]).

The problems are only potential and theoretical, and won't bite anyone
until it occurs though, and it's more likely to happen with pip/wheel than
with conda.

But anyways, this idea is still nice. I could imagine at least in arrow's
Python-C-API, there would be a

PyObject* pyarrow_array_from_c_protocol(ArrayArray*);

this way the C++ APIs can be avoided while still allowing arrays to be
created in C/C++ and used in python.

[1] https://github.com/tensorflow/tensorflow/issues/23561

Regards
>
> Antoine.
>
>
> Le 19/09/2019 à 18:21, Zhuo Peng a écrit :
> > Hi Antoine,
> >
> > I'm also interested in a stable ABI (previously I posted on this mailing
> > list about the ABI issues I had [1]). Does having such an ABI-stable
> > C-struct imply that there will be a set of C-APIs exposed by the Arrow
> > (C++) library (which I think would lead to a solution to all the inherit
> > ABI issues caused by C++)?
> >
> > [1]
> >
> https://lists.apache.org/thread.html/27b6e2a30cf93c5f5f78de970c68c7d996f538d94ab61431fa342f41@%3Cdev.arrow.apache.org%3E
> >
> > On Thu, Sep 19, 2019 at 1:07 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
> >>> I like the idea of a stable ABI for in-processing  that can be used for
> >> in
> >>> process communication.  For instance, there was a recent question on
> >>> stack-overflow on how to solve this [1].
> >>>
> >>> A couple of thoughts/questions:
> >>> * Would ArrowArray also need a self reference for children arrays?
> >>
> >> Yes, I forgot that.  I also think we don't need a separate Buffer
> >> struct, instead the Array struct should own all its buffers.
> >>
> >>> * Should transferring key-value metadata be in scope?
> >>
> >> Yes.  It could either be in the format string or a separate string.  The
> >> upside of a separate string is that a consumer may ignore it trivially
> >> if it doesn't need the information.
> >>
> >> Another open question is for nested types: does the format string
> >> represent the entire type including children?  Or must child types be
> >> read in the child arrays?  If we mimick ArrayData, then the format
> >> string should represent the entire type; it will then be more complex to
> >> parse.
> >>
> >> We should also make sure that extension types fit in the protocol.
> >>
> >>> * Should the API more closely align the IPC spec (pass a schema
> >> separately
> >>> and list of buffers instead of individual arrays)?
> >>
> >> Then you have that's not immediately usable (you have to do some
> >> processing to reconstitute the individual arrays).  One goal here is to
> >> minimize implementation costs for producers and consumers.  The
> >> assumption is a data model similar to the C++ ArrowData model; do we
> >> have implementations that use an entirely different model?  Perhaps I
> >> should take a look :-)
> >>
> >> Note that the draft I posted only concerns arrays.  We may also want to
> >> have a C struct for batches or tables.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>>
> >>> Thanks,
> >>> Micah
> >>>
> >>> [1]
> >>>
> >>
> https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
> >>>
> >>> On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org>
> >> wrote:
> >>>
> >>>>
> >>>> Hello,
> >>>>
> >>>> One thing that was discussed in the sync call is the ability to easily
> >>>> pass arrays at runtime between Arrow implementations or
> Arrow-supporting
> >>>> libraries in the same process, without bearing the cost of linking to
> >>>> e.g. the C++ Arrow library.
> >>>>
> >>>> (for example: "Duckdb wants to provide an option to return Arrow data
> of
> >>>> result sets, but they don't like having Arrow as a dependency")
> >>>>
> >>>> One possibility would be to define a C-level protocol similar in
> spirit
> >>>> to the Python buffer protocol, which some people may be familiar with
> >> (*).
> >>>>
> >>>> The basic idea is to define a simple C struct, which is ABI-stable and
> >>>> describes an Arrow away adequately.  The struct can be
> stack-allocated.
> >>>> Its definition can also be copied in another project (or interfaced
> with
> >>>> using a C FFI layer, depending on the language).
> >>>>
> >>>> There is no formal proposal, this message is meant to stir the
> >> discussion.
> >>>>
> >>>> Issues to work out:
> >>>>
> >>>> * Memory lifetime issues: where Python simply associates the Py_buffer
> >>>> with a PyObject owner (a garbage-collected Python object), we need
> >>>> another means to control lifetime of pointed areas.  One simple
> >>>> possibility is to include a destructor function pointer in the
> protocol
> >>>> struct.
> >>>>
> >>>> * Arrow type representation.  We probably need some kind of "format"
> >>>> mini-language to represent Arrow types, so that a type can be
> described
> >>>> using a `const char*`.  Ideally, primitives types at least should be
> >>>> trivially parsable.  We may take inspiration from Python here
> (`struct`
> >>>> module format characters, PEP 3118 format additions).
> >>>>
> >>>> Example C struct definition (not a formal proposal!):
> >>>>
> >>>> struct ArrowBuffer {
> >>>>   void* data;
> >>>>   int64_t nbytes;
> >>>>   // Called by the consumer when it doesn't need the buffer anymore
> >>>>   void (*release)(struct ArrowBuffer*);
> >>>>   // Opaque user data (for e.g. the release callback)
> >>>>   void* user_data;
> >>>> };
> >>>>
> >>>> struct ArrowArray {
> >>>>   // Type description
> >>>>   const char* format;
> >>>>   // Data description
> >>>>   int64_t length;
> >>>>   int64_t null_count;
> >>>>   int64_t n_buffers;
> >>>>   // Note: this pointers are probably owned by the ArrowArray struct
> >>>>   // and will be released and free()ed by the release callback.
> >>>>   struct BufferDescriptor* buffers;
> >>>>   struct ArrowDescriptor* dictionary;
> >>>>   // Called by the consumer when it doesn't need the array anymore
> >>>>   void (*release)(struct ArrowArrayDescriptor*);
> >>>>   // Opaque user data (for e.g. the release callback)
> >>>>   void* user_data;
> >>>> };
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> (*) For the record, the reference for the Python buffer protocol:
> >>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> >>>> and its C struct definition:
> >>>>
> >>
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >>>>
> >>>> Regards
> >>>>
> >>>> Antoine.
> >>>>
> >>>
> >>
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

No, the plan for this proposal is to avoid providing a C API.  Each
Arrow implementation could produce and consume the C data protocol, for
example the C++ Array class could add these methods:

class Array {
  // ...

 public:
  // Export array to the C data protocol
  void Share(ArrowArray* out);
  // Import a C data protocol array
  static Status FromShared(ArrowArray* input,
                           std::shared_ptr<Array>* out);
};

Also, I don't know why a C API exposed by the C++ library would solve
your problem.  You would still have a problem with bundling the .so,
symbol conflicts if several libraries load libarrow.so, etc.

Regards

Antoine.


Le 19/09/2019 à 18:21, Zhuo Peng a écrit :
> Hi Antoine,
> 
> I'm also interested in a stable ABI (previously I posted on this mailing
> list about the ABI issues I had [1]). Does having such an ABI-stable
> C-struct imply that there will be a set of C-APIs exposed by the Arrow
> (C++) library (which I think would lead to a solution to all the inherit
> ABI issues caused by C++)?
> 
> [1]
> https://lists.apache.org/thread.html/27b6e2a30cf93c5f5f78de970c68c7d996f538d94ab61431fa342f41@%3Cdev.arrow.apache.org%3E
> 
> On Thu, Sep 19, 2019 at 1:07 AM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
>>> I like the idea of a stable ABI for in-processing  that can be used for
>> in
>>> process communication.  For instance, there was a recent question on
>>> stack-overflow on how to solve this [1].
>>>
>>> A couple of thoughts/questions:
>>> * Would ArrowArray also need a self reference for children arrays?
>>
>> Yes, I forgot that.  I also think we don't need a separate Buffer
>> struct, instead the Array struct should own all its buffers.
>>
>>> * Should transferring key-value metadata be in scope?
>>
>> Yes.  It could either be in the format string or a separate string.  The
>> upside of a separate string is that a consumer may ignore it trivially
>> if it doesn't need the information.
>>
>> Another open question is for nested types: does the format string
>> represent the entire type including children?  Or must child types be
>> read in the child arrays?  If we mimick ArrayData, then the format
>> string should represent the entire type; it will then be more complex to
>> parse.
>>
>> We should also make sure that extension types fit in the protocol.
>>
>>> * Should the API more closely align the IPC spec (pass a schema
>> separately
>>> and list of buffers instead of individual arrays)?
>>
>> Then you have that's not immediately usable (you have to do some
>> processing to reconstitute the individual arrays).  One goal here is to
>> minimize implementation costs for producers and consumers.  The
>> assumption is a data model similar to the C++ ArrowData model; do we
>> have implementations that use an entirely different model?  Perhaps I
>> should take a look :-)
>>
>> Note that the draft I posted only concerns arrays.  We may also want to
>> have a C struct for batches or tables.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>>
>>> Thanks,
>>> Micah
>>>
>>> [1]
>>>
>> https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
>>>
>>> On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org>
>> wrote:
>>>
>>>>
>>>> Hello,
>>>>
>>>> One thing that was discussed in the sync call is the ability to easily
>>>> pass arrays at runtime between Arrow implementations or Arrow-supporting
>>>> libraries in the same process, without bearing the cost of linking to
>>>> e.g. the C++ Arrow library.
>>>>
>>>> (for example: "Duckdb wants to provide an option to return Arrow data of
>>>> result sets, but they don't like having Arrow as a dependency")
>>>>
>>>> One possibility would be to define a C-level protocol similar in spirit
>>>> to the Python buffer protocol, which some people may be familiar with
>> (*).
>>>>
>>>> The basic idea is to define a simple C struct, which is ABI-stable and
>>>> describes an Arrow away adequately.  The struct can be stack-allocated.
>>>> Its definition can also be copied in another project (or interfaced with
>>>> using a C FFI layer, depending on the language).
>>>>
>>>> There is no formal proposal, this message is meant to stir the
>> discussion.
>>>>
>>>> Issues to work out:
>>>>
>>>> * Memory lifetime issues: where Python simply associates the Py_buffer
>>>> with a PyObject owner (a garbage-collected Python object), we need
>>>> another means to control lifetime of pointed areas.  One simple
>>>> possibility is to include a destructor function pointer in the protocol
>>>> struct.
>>>>
>>>> * Arrow type representation.  We probably need some kind of "format"
>>>> mini-language to represent Arrow types, so that a type can be described
>>>> using a `const char*`.  Ideally, primitives types at least should be
>>>> trivially parsable.  We may take inspiration from Python here (`struct`
>>>> module format characters, PEP 3118 format additions).
>>>>
>>>> Example C struct definition (not a formal proposal!):
>>>>
>>>> struct ArrowBuffer {
>>>>   void* data;
>>>>   int64_t nbytes;
>>>>   // Called by the consumer when it doesn't need the buffer anymore
>>>>   void (*release)(struct ArrowBuffer*);
>>>>   // Opaque user data (for e.g. the release callback)
>>>>   void* user_data;
>>>> };
>>>>
>>>> struct ArrowArray {
>>>>   // Type description
>>>>   const char* format;
>>>>   // Data description
>>>>   int64_t length;
>>>>   int64_t null_count;
>>>>   int64_t n_buffers;
>>>>   // Note: this pointers are probably owned by the ArrowArray struct
>>>>   // and will be released and free()ed by the release callback.
>>>>   struct BufferDescriptor* buffers;
>>>>   struct ArrowDescriptor* dictionary;
>>>>   // Called by the consumer when it doesn't need the array anymore
>>>>   void (*release)(struct ArrowArrayDescriptor*);
>>>>   // Opaque user data (for e.g. the release callback)
>>>>   void* user_data;
>>>> };
>>>>
>>>> Thoughts?
>>>>
>>>> (*) For the record, the reference for the Python buffer protocol:
>>>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>>>> and its C struct definition:
>>>>
>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>
>>
> 

Re: [DISCUSS] C-level in-process array protocol

[Zhuo Peng <br...@gmail.com>]

Hi Antoine,

I'm also interested in a stable ABI (previously I posted on this mailing
list about the ABI issues I had [1]). Does having such an ABI-stable
C-struct imply that there will be a set of C-APIs exposed by the Arrow
(C++) library (which I think would lead to a solution to all the inherit
ABI issues caused by C++)?

[1]
https://lists.apache.org/thread.html/27b6e2a30cf93c5f5f78de970c68c7d996f538d94ab61431fa342f41@%3Cdev.arrow.apache.org%3E

On Thu, Sep 19, 2019 at 1:07 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
> > I like the idea of a stable ABI for in-processing  that can be used for
> in
> > process communication.  For instance, there was a recent question on
> > stack-overflow on how to solve this [1].
> >
> > A couple of thoughts/questions:
> > * Would ArrowArray also need a self reference for children arrays?
>
> Yes, I forgot that.  I also think we don't need a separate Buffer
> struct, instead the Array struct should own all its buffers.
>
> > * Should transferring key-value metadata be in scope?
>
> Yes.  It could either be in the format string or a separate string.  The
> upside of a separate string is that a consumer may ignore it trivially
> if it doesn't need the information.
>
> Another open question is for nested types: does the format string
> represent the entire type including children?  Or must child types be
> read in the child arrays?  If we mimick ArrayData, then the format
> string should represent the entire type; it will then be more complex to
> parse.
>
> We should also make sure that extension types fit in the protocol.
>
> > * Should the API more closely align the IPC spec (pass a schema
> separately
> > and list of buffers instead of individual arrays)?
>
> Then you have that's not immediately usable (you have to do some
> processing to reconstitute the individual arrays).  One goal here is to
> minimize implementation costs for producers and consumers.  The
> assumption is a data model similar to the C++ ArrowData model; do we
> have implementations that use an entirely different model?  Perhaps I
> should take a look :-)
>
> Note that the draft I posted only concerns arrays.  We may also want to
> have a C struct for batches or tables.
>
> Regards
>
> Antoine.
>
>
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
> >
> > On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Hello,
> >>
> >> One thing that was discussed in the sync call is the ability to easily
> >> pass arrays at runtime between Arrow implementations or Arrow-supporting
> >> libraries in the same process, without bearing the cost of linking to
> >> e.g. the C++ Arrow library.
> >>
> >> (for example: "Duckdb wants to provide an option to return Arrow data of
> >> result sets, but they don't like having Arrow as a dependency")
> >>
> >> One possibility would be to define a C-level protocol similar in spirit
> >> to the Python buffer protocol, which some people may be familiar with
> (*).
> >>
> >> The basic idea is to define a simple C struct, which is ABI-stable and
> >> describes an Arrow away adequately.  The struct can be stack-allocated.
> >> Its definition can also be copied in another project (or interfaced with
> >> using a C FFI layer, depending on the language).
> >>
> >> There is no formal proposal, this message is meant to stir the
> discussion.
> >>
> >> Issues to work out:
> >>
> >> * Memory lifetime issues: where Python simply associates the Py_buffer
> >> with a PyObject owner (a garbage-collected Python object), we need
> >> another means to control lifetime of pointed areas.  One simple
> >> possibility is to include a destructor function pointer in the protocol
> >> struct.
> >>
> >> * Arrow type representation.  We probably need some kind of "format"
> >> mini-language to represent Arrow types, so that a type can be described
> >> using a `const char*`.  Ideally, primitives types at least should be
> >> trivially parsable.  We may take inspiration from Python here (`struct`
> >> module format characters, PEP 3118 format additions).
> >>
> >> Example C struct definition (not a formal proposal!):
> >>
> >> struct ArrowBuffer {
> >>   void* data;
> >>   int64_t nbytes;
> >>   // Called by the consumer when it doesn't need the buffer anymore
> >>   void (*release)(struct ArrowBuffer*);
> >>   // Opaque user data (for e.g. the release callback)
> >>   void* user_data;
> >> };
> >>
> >> struct ArrowArray {
> >>   // Type description
> >>   const char* format;
> >>   // Data description
> >>   int64_t length;
> >>   int64_t null_count;
> >>   int64_t n_buffers;
> >>   // Note: this pointers are probably owned by the ArrowArray struct
> >>   // and will be released and free()ed by the release callback.
> >>   struct BufferDescriptor* buffers;
> >>   struct ArrowDescriptor* dictionary;
> >>   // Called by the consumer when it doesn't need the array anymore
> >>   void (*release)(struct ArrowArrayDescriptor*);
> >>   // Opaque user data (for e.g. the release callback)
> >>   void* user_data;
> >> };
> >>
> >> Thoughts?
> >>
> >> (*) For the record, the reference for the Python buffer protocol:
> >> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> >> and its C struct definition:
> >>
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >
>

Re: [DISCUSS] C-level in-process array protocol

[Antoine Pitrou <an...@python.org>]

Le 19/09/2019 à 09:39, Micah Kornfield a écrit :
> I like the idea of a stable ABI for in-processing  that can be used for in
> process communication.  For instance, there was a recent question on
> stack-overflow on how to solve this [1].
> 
> A couple of thoughts/questions:
> * Would ArrowArray also need a self reference for children arrays?

Yes, I forgot that.  I also think we don't need a separate Buffer
struct, instead the Array struct should own all its buffers.

> * Should transferring key-value metadata be in scope?

Yes.  It could either be in the format string or a separate string.  The
upside of a separate string is that a consumer may ignore it trivially
if it doesn't need the information.

Another open question is for nested types: does the format string
represent the entire type including children?  Or must child types be
read in the child arrays?  If we mimick ArrayData, then the format
string should represent the entire type; it will then be more complex to
parse.

We should also make sure that extension types fit in the protocol.

> * Should the API more closely align the IPC spec (pass a schema separately
> and list of buffers instead of individual arrays)?

Then you have that's not immediately usable (you have to do some
processing to reconstitute the individual arrays).  One goal here is to
minimize implementation costs for producers and consumers.  The
assumption is a data model similar to the C++ ArrowData model; do we
have implementations that use an entirely different model?  Perhaps I
should take a look :-)

Note that the draft I posted only concerns arrays.  We may also want to
have a C struct for batches or tables.

Regards

Antoine.


> 
> Thanks,
> Micah
> 
> [1]
> https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220
> 
> On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Hello,
>>
>> One thing that was discussed in the sync call is the ability to easily
>> pass arrays at runtime between Arrow implementations or Arrow-supporting
>> libraries in the same process, without bearing the cost of linking to
>> e.g. the C++ Arrow library.
>>
>> (for example: "Duckdb wants to provide an option to return Arrow data of
>> result sets, but they don't like having Arrow as a dependency")
>>
>> One possibility would be to define a C-level protocol similar in spirit
>> to the Python buffer protocol, which some people may be familiar with (*).
>>
>> The basic idea is to define a simple C struct, which is ABI-stable and
>> describes an Arrow away adequately.  The struct can be stack-allocated.
>> Its definition can also be copied in another project (or interfaced with
>> using a C FFI layer, depending on the language).
>>
>> There is no formal proposal, this message is meant to stir the discussion.
>>
>> Issues to work out:
>>
>> * Memory lifetime issues: where Python simply associates the Py_buffer
>> with a PyObject owner (a garbage-collected Python object), we need
>> another means to control lifetime of pointed areas.  One simple
>> possibility is to include a destructor function pointer in the protocol
>> struct.
>>
>> * Arrow type representation.  We probably need some kind of "format"
>> mini-language to represent Arrow types, so that a type can be described
>> using a `const char*`.  Ideally, primitives types at least should be
>> trivially parsable.  We may take inspiration from Python here (`struct`
>> module format characters, PEP 3118 format additions).
>>
>> Example C struct definition (not a formal proposal!):
>>
>> struct ArrowBuffer {
>>   void* data;
>>   int64_t nbytes;
>>   // Called by the consumer when it doesn't need the buffer anymore
>>   void (*release)(struct ArrowBuffer*);
>>   // Opaque user data (for e.g. the release callback)
>>   void* user_data;
>> };
>>
>> struct ArrowArray {
>>   // Type description
>>   const char* format;
>>   // Data description
>>   int64_t length;
>>   int64_t null_count;
>>   int64_t n_buffers;
>>   // Note: this pointers are probably owned by the ArrowArray struct
>>   // and will be released and free()ed by the release callback.
>>   struct BufferDescriptor* buffers;
>>   struct ArrowDescriptor* dictionary;
>>   // Called by the consumer when it doesn't need the array anymore
>>   void (*release)(struct ArrowArrayDescriptor*);
>>   // Opaque user data (for e.g. the release callback)
>>   void* user_data;
>> };
>>
>> Thoughts?
>>
>> (*) For the record, the reference for the Python buffer protocol:
>> https://docs.python.org/3/c-api/buffer.html#buffer-structure
>> and its C struct definition:
>> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>>
>> Regards
>>
>> Antoine.
>>
> 

Re: [DISCUSS] C-level in-process array protocol

[Micah Kornfield <em...@gmail.com>]

I like the idea of a stable ABI for in-processing  that can be used for in
process communication.  For instance, there was a recent question on
stack-overflow on how to solve this [1].

A couple of thoughts/questions:
* Would ArrowArray also need a self reference for children arrays?
* Should transferring key-value metadata be in scope?
* Should the API more closely align the IPC spec (pass a schema separately
and list of buffers instead of individual arrays)?

Thanks,
Micah

[1]
https://stackoverflow.com/questions/57966032/how-does-apache-arrow-facilitate-no-overhead-for-cross-system-communication/57967220#57967220

On Wed, Sep 18, 2019 at 10:52 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello,
>
> One thing that was discussed in the sync call is the ability to easily
> pass arrays at runtime between Arrow implementations or Arrow-supporting
> libraries in the same process, without bearing the cost of linking to
> e.g. the C++ Arrow library.
>
> (for example: "Duckdb wants to provide an option to return Arrow data of
> result sets, but they don't like having Arrow as a dependency")
>
> One possibility would be to define a C-level protocol similar in spirit
> to the Python buffer protocol, which some people may be familiar with (*).
>
> The basic idea is to define a simple C struct, which is ABI-stable and
> describes an Arrow away adequately.  The struct can be stack-allocated.
> Its definition can also be copied in another project (or interfaced with
> using a C FFI layer, depending on the language).
>
> There is no formal proposal, this message is meant to stir the discussion.
>
> Issues to work out:
>
> * Memory lifetime issues: where Python simply associates the Py_buffer
> with a PyObject owner (a garbage-collected Python object), we need
> another means to control lifetime of pointed areas.  One simple
> possibility is to include a destructor function pointer in the protocol
> struct.
>
> * Arrow type representation.  We probably need some kind of "format"
> mini-language to represent Arrow types, so that a type can be described
> using a `const char*`.  Ideally, primitives types at least should be
> trivially parsable.  We may take inspiration from Python here (`struct`
> module format characters, PEP 3118 format additions).
>
> Example C struct definition (not a formal proposal!):
>
> struct ArrowBuffer {
>   void* data;
>   int64_t nbytes;
>   // Called by the consumer when it doesn't need the buffer anymore
>   void (*release)(struct ArrowBuffer*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
>
> struct ArrowArray {
>   // Type description
>   const char* format;
>   // Data description
>   int64_t length;
>   int64_t null_count;
>   int64_t n_buffers;
>   // Note: this pointers are probably owned by the ArrowArray struct
>   // and will be released and free()ed by the release callback.
>   struct BufferDescriptor* buffers;
>   struct ArrowDescriptor* dictionary;
>   // Called by the consumer when it doesn't need the array anymore
>   void (*release)(struct ArrowArrayDescriptor*);
>   // Opaque user data (for e.g. the release callback)
>   void* user_data;
> };
>
> Thoughts?
>
> (*) For the record, the reference for the Python buffer protocol:
> https://docs.python.org/3/c-api/buffer.html#buffer-structure
> and its C struct definition:
> https://github.com/python/cpython/blob/v3.7.4/Include/object.h#L181-L195
>
> Regards
>
> Antoine.
>