You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2016/08/16 04:38:20 UTC

[jira] [Created] (ARROW-262) [Format] Add a new format document for metadata and logical types for messaging and IPC / on-wire/file representations

Wes McKinney created ARROW-262:
----------------------------------

             Summary: [Format] Add a new format document for metadata and logical types for messaging and IPC / on-wire/file representations
                 Key: ARROW-262
                 URL: https://issues.apache.org/jira/browse/ARROW-262
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Format
            Reporter: Wes McKinney
            Assignee: Wes McKinney


The existing document

https://github.com/apache/arrow/blob/master/format/Layout.md

Only describes the physical layout of fixed-size, variable-size, and other nested types (struct, union)

Meanwhile, we have begun drafting Flatbuffers IDL for Arrow metadata:

https://github.com/apache/arrow/blob/master/format/Message.fbs

I will add a document that will, to begin with:

* Explain the mapping between logical types in the metadata. For example,  definitions of important data types: integers, floating point, boolean, string (UTF-8) and binary

* Where relevant, describing how each logical type's physical memory is converted to metadata for messaging purposes (e.g. the {{RecordBatch}} concept in the IDL)

We have already begun prototype implementations in the C++ codebase (https://github.com/apache/arrow/tree/master/cpp/src/arrow/ipc) so this will serve as implementation-agnostic documentation.

Subsequently, I will make a follow up patch for discussion to hopefully address metadata shortfall between the canonical Arrow metadata and the similar metadata used by the bespoke Feather format (https://github.com/wesm/feather/blob/master/cpp/src/feather/metadata.fbs)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)