You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pstrzelczak (via GitHub)" <gi...@apache.org> on 2025/04/25 09:40:06 UTC
[I] Add an option to RecordBatchReader to infer schema for each record batch fetched separately [arrow]
pstrzelczak opened a new issue, #46228:
URL: https://github.com/apache/arrow/issues/46228
### Describe the enhancement requested
Right now `RecordBatchReader` infers schema based on the first batch. As a consequence, if any cell in the remaining stream does not comply to the schema fetching the data fails. It would be great to have an option to infer schema for each individual chunk instead of failing.
This is split from https://github.com/apache/arrow/issues/17278.
### Component(s)
Python, C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add an option to RecordBatchReader to infer schema for each record batch fetched separately [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46228:
URL: https://github.com/apache/arrow/issues/46228#issuecomment-4914450269
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add an option to RecordBatchReader to infer schema for each record batch fetched separately [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #46228: [C++] Add an option to RecordBatchReader to infer schema for each record batch fetched separately
URL: https://github.com/apache/arrow/issues/46228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Add an option to RecordBatchReader to infer schema for each record batch fetched separately [arrow]
Posted by "pstrzelczak (via GitHub)" <gi...@apache.org>.
pstrzelczak commented on issue #46228:
URL: https://github.com/apache/arrow/issues/46228#issuecomment-3048118009
I managed to work around the lack of this feature to infer the schema reliably in a single pass by writing and reading in-memory csv file for each record batch:
```
schemas = []
with pyarrow.csv.open_csv(csv_path) as input_csv:
for batch in input_csv:
buffer = io.BytesIO()
pyarrow.csv.write_csv(batch, buffer)
buffer.seek(0)
inferred_table = pyarrow.csv.read_csv(buffer)
current_chunk_schema = inferred_table.schema
if current_chunk_schema not in schemas:
schemas.append(current_chunk_schema)
```
and then resolve multiple schemas using pyarrow.unify_schemas
Still, it would be great to avoid additional steps and use (not available now) the option to infer schema for each individual chunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org