Arrow Format vs Feather v2

[Daniel Nugent <nu...@gmail.com>]

Was just reading the 0.17 release notes (congratulations to the maintainers, btw), and was wondering if there could be some clarification on the language about file formats.

The notes mention that the compression support available for Feather 2 will be formalized in the Arrow format at a later time.

Does that mean that they will be formalized for in-memory and on the wire Arrow messages? Or that there will be another, separate from Feather 2, on-disk representation for Arrow called “Arrow file format” or something along those lines?

Thanks,

-Dan Nugent

Re: Arrow Format vs Feather v2

[Wes McKinney <we...@gmail.com>]

hi Dan

See https://lists.apache.org/thread.html/r0be397a5f901b9dc8787a7dbcb0a34c9ed60ad07ff1e3f064d418a98%40%3Cdev.arrow.apache.org%3E.
There is an experimental implementation in C++ which is being used for
"Feather V2"

The Arrow specification does include a "file" format -- this is
exactly what "Feather V2" is using

https://github.com/apache/arrow/blob/master/docs/source/format/Columnar.rst#ipc-file-format

- Wes

On Thu, Apr 23, 2020 at 4:20 PM Daniel Nugent <nu...@gmail.com> wrote:
>
> Was just reading the 0.17 release notes (congratulations to the maintainers, btw), and was wondering if there could be some clarification on the language about file formats.
>
> The notes mention that the compression support available for Feather 2 will be formalized in the Arrow format at a later time.
>
> Does that mean that they will be formalized for in-memory and on the wire Arrow messages? Or that there will be another, separate from Feather 2, on-disk representation for Arrow called “Arrow file format” or something along those lines?
>
> Thanks,
>
> -Dan Nugent