You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jonded94 (via GitHub)" <gi...@apache.org> on 2025/05/12 13:30:49 UTC
[I] [Python] Files containing binary data with >=8_388_855 bytes written with `arrow-rs` can't be read with `pyarrow` [arrow]
jonded94 opened a new issue, #46404:
URL: https://github.com/apache/arrow/issues/46404
### Describe the bug, including details regarding any error messages, version, and platform.
Hello,
internally, we wrote an own library that wraps `arrow-rs` to make it useable from Python.
Such a thing also exists publicly available through `arro3` which I used here for some minimal reproducible example:
```
import pyarrow.parquet
import arro3.io
data = [8388855, 8388924, 8388853, 8388880, 8388876, 8388879]
schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
data = [{"html": b"0" * d} for d in data]
t = pyarrow.Table.from_pylist(data, schema=schema)
path = "/tmp/foo.parquet"
with open(path, "wb") as file:
for b in t.to_batches():
arro3.io.write_parquet(b, file, max_row_group_size=len(data) - 3)
reader = pyarrow.parquet.ParquetFile(path)
for i in range(2):
print(len(reader.read_row_group(i)))
```
This code writes a bit of dummy binary data through `arrow-rs`. Reading that with `pyarrow` results in
```
File "pyarrow/_parquet.pyx", line 1655, in pyarrow._parquet.ParquetReader.read_row_group
File "pyarrow/_parquet.pyx", line 1691, in pyarrow._parquet.ParquetReader.read_row_groups
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Couldn't deserialize thrift: No more data to read.
Deserializing page header failed.
```
---
### Obversations
- Reading in the same file through `arro3` or own internal library wrapping `arrow-rs` works just fine
- Reading in the same file through `duckdb` also works just fine
- Reducing the amount of binary data per row suddenly leads to the error disappearing
- Issue is reproducible with `pyarrow` version `18.1.0`, `19.0.1` and `20.0.0`
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873378936
Update: I think you may also have to update this setting (which controls the page index metadata lenght):
- [`WriterPropertiesBuilder::set_column_index_truncate_length`](https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_index_truncate_length)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] Files with very large data page header can't be read with `pyarrow` [arrow]
Posted by "ayushbansal07 (via GitHub)" <gi...@apache.org>.
ayushbansal07 commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-3382837739
Hi, I have raised a PR for this. Please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2939902694
> The only remaining issue in this problemspace now is that `pyarrow` isn't able to read parquet files with page header statistics which contain values larger than 8MiB.
I guess we could expose a setting to allow increasing the max page header size. There is already a [low-level C++ API](https://github.com/apache/arrow/blob/832bfa1ec67c3a6d43dab5abe7d84271680fee9c/cpp/src/parquet/column_reader.h#L150) for it, but it should be moved to the `ReaderProperties` class and then exposed in Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873081010
Sure, although the code snippet I already sent is very easily reproducible by just installing `arro3-io` (`pip install arro3-io`).
I just googled for random file sharing websites, hopefully one of these links works:
- https://limewire.com/d/eUky5#Mxn7Vs9n2k
- https://www.filemail.com/d/eiieexpsyxtvwus
- https://filebin.net/3zqvokkogaew8jxr/foo.parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873252315
> Ok, I looked at this a bit and the problem is that the file contains a data page with a header larger than 16 MB. (probably because non-truncated statistics are written out). While technically valid, this is not desirable and arrow-rs should probably avoid producing such files. @alamb
> So you think setting [this](https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_statistics_truncate_length) parameter to something below 16MiB should solve the issue? (if yes, probably this should be the default on the arrow-rs side, right?)
I think you should set that parameter to something much lower like 128 as it refers to the maximum size of each value, not the size of the entire page (which is what it sounds like parquet-cpp is limited with 16MB).
I did notice that the default value is not well documented and the default is actually `None` (no truncation). I will file a ticket to consider a different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873013274
Thanks for the report @jonded94 . Is it possible for you to upload the demonstration file somewhere for ease of reproduction?
(you can perhaps enable compression under Parquet if that still allows reproducing)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2939736293
@pitrou fyi, the issue in `arrow-rs` that page headers can't be truncated is now fixed (https://github.com/apache/arrow-rs/issues/7489). The only remaining issue in this problemspace now is that `pyarrow` isn't able to read parquet files with page header statistics which contain values larger than 8MiB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873175632
Ok, I looked at this a bit and the problem is that the file contains a data page with a **header** larger than 16 MB. While technically valid, this is pathological and `arrow-rs` should probably avoid producing such files. @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "jonded94 (via GitHub)" <gi...@apache.org>.
jonded94 commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873236523
Thank you for your extremely quick investigation!
- So you think setting [this](https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_statistics_truncate_length) parameter to something below 16MiB should solve the issue? (if yes, probably this should be the default on the `arrow-rs` side, right?)
- Since other libraries (duckdb, arrow-rs, ...) are able to read the parquet file, do you think it could be valuable if `[py]arrow` also were able to read it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873114453
cc @mapleFU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873252881
> * So you think setting [this](https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_statistics_truncate_length) parameter to something below 16MiB should solve the issue? (if yes, probably this should be the default on the `arrow-rs` side, right?)
Yes, it definitely should.
> * Since other libraries (duckdb, arrow-rs, ...) are able to read the parquet file, do you think it could be valuable if `[py]arrow` also were able to read it?
Yes, it would. We could definitely relax the current limits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Files containing binary data with >=8_388_855 bytes per row written with `arrow-rs` can't be read with `pyarrow` [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #46404:
URL: https://github.com/apache/arrow/issues/46404#issuecomment-2873270229
Filed a ticket to discuss a change in default statistics truncation in arrow-rs:
- https://github.com/apache/arrow-rs/issues/7490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org