You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Rob Ambalu (JIRA)" <ji...@apache.org> on 2018/05/14 15:24:00 UTC
[jira] [Created] (ARROW-2579) Appending to stremable table file
format doesnt seem to work
Rob Ambalu created ARROW-2579:
---------------------------------
Summary: Appending to stremable table file format doesnt seem to work
Key: ARROW-2579
URL: https://issues.apache.org/jira/browse/ARROW-2579
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 0.9.0
Reporter: Rob Ambalu
As far as I can tell it looks like appending to a streaming file format isn’t currently supported, is that right?
RecordBatchStreamWriter always writes the schema up front, and it doesn’t look like a schema is expected mid file ( assuming im doing this append test correctly, this is the error I hit when I try to read back this file into python:
Traceback (most recent call last):
File "/home/ra7293/rba_arrow_mmap.py", line 9, in <module>
table = reader.read_all()
File "ipc.pxi", line 302, in pyarrow.lib._RecordBatchReader.read_all
File "error.pxi", line 79, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Message not expected type: record batch, was: 1
This reader script works fine if I write once / don’t append.
Seeing as IO interfaces support Append, streaming should support it as well ( if for whatever reason this cant be supported, RecordBatchStreamWriter should throw if configured with an OutputStreamer that is attempting to append )
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)