You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "progval (via GitHub)" <gi...@apache.org> on 2025/01/16 11:32:15 UTC
[I] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
progval opened a new issue, #45283:
URL: https://github.com/apache/arrow/issues/45283
### Describe the bug, including details regarding any error messages, version, and platform.
Since pyarrow v19, this file cannot be read anymore:
[test.parquet.gz](https://github.com/user-attachments/files/18438726/test.parquet.gz) (decompress it with `gunzip`; I had to compress it for Github to accept the upload)
with pyarrow 18.1.0:
```
$ python3
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.dataset
>>> dataset = pyarrow.dataset.dataset("test.parquet", format="parquet")
>>> dataset.to_table().to_pylist()
[{'id': 0, 'type': 'ori', 'sha1_git': b'\x8fP\xd3\xf6\x0e\xae7\r\xdb\xf8\\\x86!\x9cU\x10\x8a5\x01e'}, {'id': 2, 'type': 'ori', 'sha1_git': b'\x83@O\x99Q\x18\xbd%wOJ\xc1D"\xa8\xf1u\xe7\xa0T'}, {'id': 3, 'type': 'rev', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\t'}, {'id': 4, 'type': 'rel', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10'}, {'id': 6, 'type': 'rev', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03'}, {'id': 7, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02'}, {'id': 8, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05'}, {'id': 9, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06'}, {'id': 10, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04'}, {'id': 11, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'}, {'id': 12, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08'}, {'id': 13, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x07'}, {'id': 14, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12'}, {'id': 15, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x11'}, {'id': 16, 'type': 'rev', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x13'}, {'id': 17, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16'}, {'id': 18, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15'}, {'id': 19, 'type': 'rel', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00!'}, {'id': 20, 'type': 'rev', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18'}, {'id': 21, 'type': 'rel', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x19'}, {'id': 22, 'type': 'dir', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x17'}, {'id': 23, 'type': 'cnt', 'sha1_git': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14'}]
>>>
```
with pyarrow 19.0.0:
```
$ python3
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.dataset
>>> dataset = pyarrow.dataset.dataset("test.parquet", format="parquet")
>>> dataset.to_table().to_pylist()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow/_dataset.pyx", line 574, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3865, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Repetition level histogram size mismatch
```
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2595902840
Let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2610954388
@h-vetinari: A patch release is being discussed currently [on Zulip](https://ursalabs.zulipchat.com/#narrow/channel/180245-dev/topic/Arrow.2019.2E0.2E1.20release.20-.20is.20it.20necessary.3F) and it looks like the agreement is to do a 19.0.1 to include this fix.
And in case it helps anyone, here's a minimal reproduction of Rust code to generate a Parquet file (Py)Arrow 19.0.0 can't read:
```rust
// Cargo.toml
//
// [package]
// name = "example"
// version = "0.1.0"
// edition = "2021"
// [dependencies]
// arrow = "53.0.0"
// parquet = "53.0.0"
use std::{fs::File, sync::Arc};
use arrow::array::{ArrayRef, RecordBatch, StringArray};
use parquet::{arrow::ArrowWriter, basic::Compression, file::properties::WriterProperties};
fn main() {
let strings = StringArray::from(vec!["a", "b", "c", "d"]);
let batch = RecordBatch::try_from_iter(vec![
("letters", Arc::new(strings) as ArrayRef),
]).unwrap();
let file = File::create("../rust.parquet").expect("Failed to create file.");
let props = WriterProperties::builder()
.set_compression(Compression::SNAPPY)
.build();
let mut writer = ArrowWriter::try_new(file, batch.schema(), Some(props)).unwrap();
writer.write(&batch).expect("Writing batch");
writer.close().unwrap();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "RickLeite (via GitHub)" <gi...@apache.org>.
RickLeite commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2643041522
Any workaround with Delta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2611345970
Just an update: 19.0.1 is planned. I've sent [an email](https://lists.apache.org/thread/rcx9pj55g1rqs6x8jdsms1xlgxf63s6l) to the dev list with the plan and there's now [a milestone](https://github.com/apache/arrow/milestone/68) for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2672763584
Just an update: [19.0.1](https://arrow.apache.org/blog/2025/02/16/19.0.1-release/) is now out and fixes this issue. If you're still running into this after upgrading, please feel free to file an issue at https://github.com/apache/arrow/issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "FayeSpica (via GitHub)" <gi...@apache.org>.
FayeSpica commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-3405010940
> [@vadimkantorov](https://github.com/vadimkantorov) there was a patch release (19.0.1) to fix this issue with a handful of bug fixes, should be completely compatible with 19.0.0. Can you try that one?
<img width="887" height="162" alt="Image" src="https://github.com/user-attachments/assets/be6bfc94-96c0-40e0-9124-82d8cb285a57" />
<img width="604" height="213" alt="Image" src="https://github.com/user-attachments/assets/48803a2e-966c-4ccd-956f-4c395bd62955" />
```shell
python -m pip install pyarrow==19.0.1
```
It works!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "ldacey (via GitHub)" <gi...@apache.org>.
ldacey commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2604977438
> Do you know how the parquet file was generated? This is related to the newly implemented size statistics: https://github.com/raulcd/arrow/blob/f93004f23f7cb1a641abb805b10fb845c77bb23f/cpp/src/parquet/size_statistics.cc#L57-L60
>
> cc [@wgtmac](https://github.com/wgtmac)
I ran into this issue with parquet files written by the delta-rs library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2595321907
Do you know how the parquet file was generated? This is related to the newly implemented size statistics:
https://github.com/raulcd/arrow/blob/f93004f23f7cb1a641abb805b10fb845c77bb23f/cpp/src/parquet/size_statistics.cc#L57-L60
cc @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "vadimkantorov (via GitHub)" <gi...@apache.org>.
vadimkantorov commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-3234093689
We stumbled on a similar problem: a parquet file created using pyarrow 21.0 could not be opened on 19.0 with this error. Is it this same bug? Or a different one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "progval (via GitHub)" <gi...@apache.org>.
progval commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2595598718
With version 53.3.0 of the Rust [`parquet` crate](https://lib.rs/crates/parquet).
My code to generate it is: https://gitlab.softwareheritage.org/swh/devel/swh-graph/-/blob/master/tools/provenance/src/bin/list-provenance-nodes.rs?ref_type=heads
I can work on a smaller repro code if you think that's helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2595693825
From my understanding parquet-rs builds statistics by default so now that we are processing those on parquet-cpp you might have found an incompatibility issue. I guess the issue stops failing if you disable statistics on the Writer, right? could you validate that please?
https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterProperties.html#method.statistics_enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "progval (via GitHub)" <gi...@apache.org>.
progval commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2595748372
Confirmed, this happens both with `EnabledStatistics::Page` and `EnabledStatistics::Chunk`. Specifically, this happens iff they are enabled on the `type` column which is [defined as type `Dictionary(Int8.into(), Utf8.into()), false)`](https://gitlab.softwareheritage.org/swh/devel/swh-graph/-/blob/0a8abbe64ac1e436caf3ebe1d12a0b4f6b89c761/tools/provenance/src/node_dataset.rs#L24)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #45283: [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0
URL: https://github.com/apache/arrow/issues/45283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-3234392778
@vadimkantorov there was a patch release (19.0.1) to fix this issue with a handful of bug fixes, should be completely compatible with 19.0.0. Can you try that one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2608719730
> @raulcd: I marked this as a possible backport-candidate, even though, I don't think there are current plans for a patch or a minor release for v19
Given the severity of the issue, it seems that a patch release must happen regardless? Otherwise people get the following idea (somewhat justifiably):
> We probably don't want [an upgrade] as arrow 19.0.0 is broken across the ecosystem due to incompatibility with `arrow-rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2604151640
Issue resolved by pull request 45285
https://github.com/apache/arrow/pull/45285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++][Parquet] "OSError: Repetition level histogram size mismatch" when reading parquet file in pyarrow since 19.0.0 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45283:
URL: https://github.com/apache/arrow/issues/45283#issuecomment-2604217796
@wgtmac I marked this as a possible backport-candidate, even though, I don't think there are current plans for a patch or a minor release for v19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org