You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Nong Li (JIRA)" <ji...@apache.org> on 2017/01/11 01:02:47 UTC
[jira] [Created] (ARROW-474) Create an Arrow streaming file fomat
Nong Li created ARROW-474:
-----------------------------
Summary: Create an Arrow streaming file fomat
Key: ARROW-474
URL: https://issues.apache.org/jira/browse/ARROW-474
Project: Apache Arrow
Issue Type: Improvement
Reporter: Nong Li
It would be helpful to have a file/container layout that allows for streaming consumer and producers of arrow data.
As a motivating example, a reader could have this API:
Iterator<Record> read(java.io.InputStream source).
Similar a writer can have:
void write(RecordBatch, java.io.OutputStream dest)
The streams can be from a variety of sources: it be files, over the network, shared memory, pipe, etc. The layout would make it possible to implement
both APIs without requiring intermediate buffering.
Speaking with Julien, this involves creating a simple container format similar to ARROW-264 but suitable for streaming consumer and producers. The layout in ARROW-264 has the schema at the end so can't be used in a streaming way. This format would likely be a rearranging of the current File layout pieces, something similar to:
MAGIC, schema, totalBatches, (RecordBatchHeader, RecordBatchBody)*, MAGIC (or some end marker)
totalBatches is optional for streaming producers that don't know.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)