You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ben-freist (via GitHub)" <gi...@apache.org> on 2025/07/22 10:54:04 UTC
[I] Error when reading row group larger than 2GB [arrow-rs]
ben-freist opened a new issue, #7973:
URL: https://github.com/apache/arrow-rs/issues/7973
**Describe the bug**
I can write a table with a column whose entries together are larger than 2**31-1.
Reading them fails with an overflow error.
**To Reproduce**
```
from pathlib import Path
import pyarrow
import tempfile
import arro3.io
schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
sizes = [5068563] * 500
new_table = pyarrow.Table.from_pylist([{"html": b"0" * size} for size in sizes], schema)
with tempfile.TemporaryDirectory() as tmp_dir:
tmp = Path(tmp_dir) / "evil.parquet"
arro3.io.write_parquet(new_table, tmp)
print(len(arro3.io.read_parquet(tmp).read_all()))
```
This throws the following error:
```
Traceback (most recent call last):
File "overflow.py", line 87, in <module>
print(len(arro3.io.read_parquet(tmp).read_all()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: C Data interface error: Parquet argument error: Parquet error: index overflow decoding byte array
```
**Expected behavior**
Should print 500.
**Additional context**
arro3 is just a thin wrapper around arro-rs, which I'm using for convenience.
I'm using arro3-io version 0.5.1 and python 3.12.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "JigaoLuo (via GitHub)" <gi...@apache.org>.
JigaoLuo commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3410458370
I see the same error during my parquet rewriting today with `arrow-array-56.1.0`
```
thread '<unnamed>' panicked at /home/jluo/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.1.0/src/builder/generic_bytes_builder.rs:86:57:
byte array offset overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at /home/jluo/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.1.0/src/builder/generic_bytes_builder.rs:86:57:
byte array offset overflow
```
(the same error pattern then repeats)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3853234125
> My proposed approach is
Yes this approach sounds good.
> but instead to have the Parquet decoder emit smaller RecordBatches when the target Arrow array would exceed representable limits, similar to how pyarrow internally splits row groups.
Yes this is my understanding too
> Add a regression test in arrow-rs that reproduces the current overflow panic when reading a Parquet file with large string/binary data using default settings.
I think the code paths will be different for different string encodings, so I recommend you also add tests explicitly for the string Encoings: https://docs.rs/parquet/latest/parquet/basic/enum.Encoding.html
* PLAIN encoding
* DELTA_LENGTH_BYTE_ARRAY
* DELTA_BYTE_ARRAY
* RLE_DICTIONARY
> Introduce logic at the Parquet reader layer to stop decoding early and emit a partial RecordBatch when adding more rows would overflow the offset buffer.
Yes, this sounds good too. I think the key will be to ensure adding the check doesn't slow down the existing reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3491293286
Thanks @jonded94
I think the workaround is to read the data as a LargeStringArray or StringView (`DataType::LargeUtf8` or `DataType::StringView`) or crank down the batch size
The real solution would be to teach the parquet decoder how to emit batches smaller than `batch_size` when it hit such an issue
> (using our own PyO3 wrapper layer around arrow-rs here)
Can you point me at that? Maybe we should change the default type read by arrow-rs for Strings to StringView 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3853537062
Thanks @alamb for the confirmation!
I’ll start by adding regression tests that reproduce the overflow for large string/binary columns, and I’ll include coverage for the different Parquet string encodings (PLAIN, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, and RLE_DICTIONARY).
Once the tests are in place, I’ll move on to exploring a minimal batching change at the Parquet reader layer that avoids offset overflows without impacting the common path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-5836236942
Related issue:
- https://github.com/apache/arrow-rs/issues/6946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3492011640
> > (using our own PyO3 wrapper layer around arrow-rs here)
>
> Can you point me at that?
Sadly no, since that is a company-internal library 😅 But it's in principle very similar to `arro3` (https://github.com/kylebarron/arro3) or even directly inspired by it in parts.
> I think the workaround is to read the data as a LargeStringArray or StringView (DataType::LargeUtf8 or DataType::StringView)
> [...]
> Maybe we should change the default type read by arrow-rs for Strings to StringView 🤔
I get how `LargeUtf8` would solve the issue, but could you briefly explain how `StringView` works and how it's able to solve the 32-bit overflow issue? I'm familiar with the concept of string views in C++, but I don't quite get what it would do in Arrow.
Slightly tangential: In this particular issue, I think `pyarrow.ParquetFile(...).read_row_group(0)` wouldn't crash because it's returning a `Table` which internally is representing the columns as `ChunkedArray` instead of `Array`, i.e. it's circumventing the issue that a single `StringArray` has to point to more than `2**32` bytes of String data by just splitting the arrays up.
As `arrow-rs` does not have the concept of "ChunkedArray" or "Table", we can only ever read as a single RecordBatch here. Would a single `StringView` RecordBatch be able to make more than `2**32` bytes of string data available? Wouldn't the string view still need a pointer larger than 32-bit to access all data?
Also another stupid question, sorry: How/where can you set with which type a specific column shall be read in `arrow-rs`? We have some schema upcasting logic in our internal library, but we apply that *after* the data was read from `arrow-rs`, so it actually didn't help us here since we already error out from stuff happening in `arrow-rs` first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3102539987
Just for reference, we're seeing the same error even with a `arrow-rs` built from the most recent git commit https://github.com/apache/arrow-rs/commit/291e6e575c727a98ee52b617da0c8de64a821e09 (dev commit after Arrow 55.2.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3853037835
Hi all,
I’ve gone through the full discussion, and I agree this is an arrow-rs Parquet reader issue: the reader currently attempts to materialize batch_size rows into a single Utf8/binary array, which can overflow 32-bit offsets when the total string/binary data exceeds 2GB.
From my understanding, the correct fix is not to require schema changes (e.g., LargeUtf8 or StringView) or smaller user-provided batch sizes, but instead to have the Parquet decoder emit smaller RecordBatches when the target Arrow array would exceed representable limits, similar to how pyarrow internally splits row groups.
My proposed approach is
1. Add a regression test in arrow-rs that reproduces the current overflow panic when reading a Parquet file with large string/binary data using default settings.
2. Trace the Parquet Arrow decoding path to identify where "batch_size" rows are accumulated into a single "StringArray".
3. Introduce logic at the Parquet reader layer to stop decoding early and emit a partial RecordBatch when adding more rows would overflow the offset buffer.
4. Continue decoding the remaining rows in subsequent RecordBatches, treating batch_sizes as a target rather than a hard requirement.
5. Ensure this behavior only triggers when necessary so there is no impact on common cases.
I plan to start with step (1) by adding the failing test, then iterate on the batching logic once the failure is covered.
Does this approach align with the intended direction for fixing this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3492435673
Just leaving this small investigation of what `pyarrow` does here (it indeed splits up the row group into smaller batches and if one were to try concatenating that, we get the same error):
```
>>> import pyarrow.parquet
>>> file = pyarrow.parquet.ParquetFile("000_00000.parquet")
>>> file.metadata
<pyarrow._parquet.FileMetaData object at 0x7f28f8537b00>
created_by: parquet-cpp-arrow version 19.0.0
num_columns: 8
num_rows: 776000
num_row_groups: 1
format_version: 2.6
serialized_size: 2665
>>> rg = file.read_row_group(0)
>>> rg
pyarrow.Table
text: string
id: string
edu_int_score: int64
edu_score: double
fasttext_score: double
language: string
language_score: double
url: string
>>> rg.num_rows
776000
>>> batches = list(rg.to_batches())
>>> len(batches)
3
>>> [b.num_rows for b in batches]
[351429, 355189, 69382]
>>> c = rg.column("text")
>>> c.combine_chunks()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/table.pxi", line 780, in pyarrow.lib.ChunkedArray.combine_chunks
File "pyarrow/array.pxi", line 5019, in pyarrow.lib.concat_arrays
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays, consider casting input from `string` to `large_string` first.
```
@alamb thanks for all your pointers! `StringView` really sounds like a cool concept!
Casting of types during read (for example `String` into `StringView`) probably should be an opt-in behaviour, right? If we anyways are fine with reading data with a type different from how it may be was stored with, we also could upcast to `LargeString` directly, couldn't we?
We probably want to have 2 solutions here, with the first one being that the `RecordBatchIterator` should intelligently lower batch size if the result wouldn't be expressable as Arrow data as you described, and the second one being this opt-in type casting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3492486369
(BTW internally the parquet reader doesn't *cast* the data -- it reads the data natively into the correct array type )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3854131064
I’ve opened a PR adding regression tests.
The PR adds test coverage for large binary columns across multiple Parquet encodings (PLAIN, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, RLE_DICTIONARY) and documents the current overflow behavior. This should help validate a follow-up fix that avoids offset overflows by emitting smaller RecordBatches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB [arrow-rs]
Posted by "ben-freist (via GitHub)" <gi...@apache.org>.
ben-freist commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3102226679
For convenience here's the file: https://limewire.com/d/oaah6#xuomvd1yxZ
I've used compression='gzip(2)' to make it a bit smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3117837677
> I've used compression='gzip(2)' to make it a bit smaller.
Here is the file hosted on gihub for convenience: [evil.zip](https://github.com/user-attachments/files/21431625/evil.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3490383054
Bumping this issue again, even appears with public data (https://huggingface.co/datasets/HuggingFaceTB/dclm-edu):
```
>>> ParquetFile("/ceph-hdd-fal/home/fmt/downloaded_data/HuggingFaceTB/dclm-edu/data/000_00001.parquet").read_row_group(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array
>>> ParquetFile("/ceph-hdd-fal/home/fmt/downloaded_data/HuggingFaceTB/dclm-edu/data/000_00002.parquet").read_row_group(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array
```
(using our own PyO3 layer around `arrow-rs` here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3119283361
The failure is here: https://github.com/apache/arrow-rs/blob/9c0cb9a56f0099e7d39087826d7e409ce0f1bf5f/parquet/src/arrow/buffer/offset_buffer.rs#L78-L77
Which is related to creating `StringArray` where the array has more than 2GB of data. With the default batch_size of 8k, rows, that means the average string length in each batch is restricted to 2GB / 8k = 256k
Your data file has a length of 5MB per row, which means you can only fit ~409 rows in each batch before exceeding the 2GB limit
The real fix for this bug is probably for the parquet decoder to internally create smaller batches when the data can't fit into the target batch size and data type
## Workaround 1: use a smaller batch size
1. Set the batch size to be smaller (it works for me with a batch size of 100)
I am able to reproduce this using datafusion like
```shell
datafusion-cli -c "select length(html) from 'evil.parquet';"
DataFusion CLI v49.0.0
thread 'main' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-55.2.0/src/builder/generic_bytes_builder.rs:86:57:
byte array offset overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Setting the batch size to 100 lets it go:
```
andrewlamb@Andrews-MacBook-Pro-3 Downloads % datafusion-cli -c "set datafusion.execution.batch_size=100; select avg(length(html)) from 'evil.parquet';"
DataFusion CLI v49.0.0
0 row(s) fetched.
Elapsed 0.000 seconds.
+------------------------------------------+
| avg(character_length(evil.parquet.html)) |
+------------------------------------------+
| 5068563.0 |
+------------------------------------------+
1 row(s) fetched.
Elapsed 0.528 seconds.
````
## Workaround 2: use a different arrow type:
You can potentially override the schema to use `DataType::LargeUt8` rather than Utf8`DataType::Utf8`: https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3998738220
I've opened a PR that implements **step 4** having the decoder emit smaller `RecordBatches` automatically rather than returning an error.
**PR:** [Parquet] Split byte-array batches transparently when i32 offset would overflow #9504
The fix makes `batch_size` a target rather than a hard limit. When the next value would cause the accumulated byte-array data to exceed `i32::MAX`, all four decoders (**Plain, DeltaLength, DeltaByteArray, and Dictionary**) now stop early and return the partial batch. The decoder's internal position is left at the unread value, so the next `read_records()` call resumes seamlessly; no rows are lost, duplicated, or reordered.
**Key changes:**
* `OffsetBuffer::would_overflow(data_len)` a new zero-cost inline helper to detect overflow before any mutation
* All four byte-array decoders updated to call `would_overflow` and break cleanly instead of propagating an error
* Unit tests added for the new helper and for partial-read behaviour
This means files like the ones from **HuggingFaceTB/dclm-edu** or the original reproduction case (`[5068563] * 500`) will be transparently readable with default settings and **no schema changes required**.
Would appreciate a review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3492484813
> Casting of types during read (for example String into StringView) probably should be an opt-in behaviour, right? If we anyways are fine with reading data with a type different from how it may be was stored with, we also could upcast to LargeString directly, couldn't we?
Yes -- you can already opt in by creating a reader using
https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderBuilder.html#method.new_with_options
and passing in a ReaderOptions setup with https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_schema
(if you explicitly specify to use `DataType::Utf8View` that is what will come out of the reader)
I had to poke around in the docs to find this -- I think it would be nice to add an example in the docs to make it easier to find. I'll try and make a PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3490389842
Bumping this issue again, appears with public data too (https://huggingface.co/datasets/HuggingFaceTB/dclm-edu).
```
>>> ParquetFile(".../dclm-edu/data/000_00001.parquet").read_row_group(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array
>>> ParquetFile(".../dclm-edu/data/000_00002.parquet").read_row_group(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Arrow read/write error: Parquet argument error: Parquet error: index overflow decoding byte array
```
(using our own PyO3 wrapper layer around `arrow-rs` here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-6335601003
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3123427817
@alamb setting DataType::LargeUt8 (which is large_string() in pyarrow) does fix it, but this still is a arrow-rs bug AFAIS, since both pyarrow and duckdb can read the file just fine. I assume they do some internal splitting into smaller batches to not run into integer overflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3124301611
> [@alamb](https://github.com/alamb) setting DataType::LargeUt8 (which is large_string() in pyarrow) does fix it, but this still is a arrow-rs bug AFAIS, since both pyarrow and duckdb can read the file just fine without changing the schema. I assume they do some internal splitting into smaller batches to not run into integer overflows.
Yes I agree this is a bug in arrow-rs (actually I think it is a bug in the parquet reader which is part of this repo)
What should happen is that when reading data into a Utf8 column, if `batch_size` records can't be read into the target StringArray without overflowing the offesets (aka more than 2GB) fewer than `batch_size` records should be read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3492066161
> I get how `LargeUtf8` would solve the issue, but could you briefly explain how `StringView` works and how it's able to solve the 32-bit overflow issue? I'm familiar with the concept of string views in C++, but I don't quite get what it would do in Arrow.
I think this blog does a pretty good explaining what is going on: https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1 (basically strings are stored as a pointer / offset with multiple buffers)
> As arrow-rs does not have the concept of "ChunkedArray" or "Table", we can only ever read as a single RecordBatch here.
I think `Vec<RecordBatch>` is the equivalent -- you certainly don't have to read the entire dataset into a single `RecordBatch`
> Would a single StringView RecordBatch be able to make more than 2**32 bytes of string data available? Wouldn't the string view still need a pointer larger than 32-bit to access all data?
Yes -- it can do more than 2^32 bytes
The way it does so is that it has multiple "buffers" (which can each be 2GB) and then stores the pointer as a buffer index (i32) and buffer offset (i32)
You can read more here
- https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3493583809
@alamb could you give me a brief opinion about a PR I sketched (https://github.com/apache/arrow-rs/pull/8790)? This PR basically adds a little convenience wrapper in `arrow-pyarrow` for exposing a `Vec<RecordBatch>` from the Rust side very easily as a `pyarrow.Table` to the Python side, to basically have feature parity with `pyarrow.ParquetFile.read_row_group`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error when reading row group larger than 2GB [arrow-rs]
Posted by "ben-freist (via GitHub)" <gi...@apache.org>.
ben-freist commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3102242636
pyarrow can read the file just fine, that's what makes me think that this might be an error in arrow-rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org