Schema serialization and bindings generation?

[Max Burke <ma...@urbanlogiq.com>]

Hello!

I had a couple questions. I have a system that is currently using
Flatbuffers because they provide a couple features: 1) flatbuffers support
generating flatbuffer-encoded representations of schemas (ie: reflection)
and 2) flatbuffers have a bindings generator for type-safe access in client
languages.

That said, the ecosystem evolving around Arrow is really intriguing.

Is there any equivalent now for serializing schemas/reflection?

Are there any plans for supporting binding generation tools?

-- 
-Max

Re: Schema serialization and bindings generation?

[Wes McKinney <we...@gmail.com>]

hi Max,

Note that Flatbuffers and the Arrow columnar format are certainly not
competing with each other. While we use Flatbuffers to serialize
schemas, data headers, and other metadata, the data itself is not
stored inside a Flatbuffer or similar serialized structure. This is a
common misconception so worth restating.

You can see the serialization details in the columnar format documentation

http://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc

Each library generally has functions to serialize schemas into
language-independent protocol messages.

Regarding "binding generation tools", as I recall there has been some
discussion in the past of generating schema-specific accessor APIs
similar to how Protobuf/Thrift/Flatbuffers generates accessor APIs,
this seems like an interesting direction to explore for a volunteer in
the community.

- Wes

On Fri, Oct 25, 2019 at 1:49 PM Max Burke <ma...@urbanlogiq.com> wrote:
>
> Hello!
>
> I had a couple questions. I have a system that is currently using Flatbuffers because they provide a couple features: 1) flatbuffers support generating flatbuffer-encoded representations of schemas (ie: reflection) and 2) flatbuffers have a bindings generator for type-safe access in client languages.
>
> That said, the ecosystem evolving around Arrow is really intriguing.
>
> Is there any equivalent now for serializing schemas/reflection?
>
> Are there any plans for supporting binding generation tools?
>
> --
> -Max