You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "TomAugspurger (via GitHub)" <gi...@apache.org> on 2025/10/28 16:11:06 UTC
[I] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
TomAugspurger opened a new issue, #47981:
URL: https://github.com/apache/arrow/issues/47981
### Describe the bug, including details regarding any error messages, version, and platform.
Something about [this parquet file](https://github.com/Parquet/parquet-compatibility/blob/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet) form https://github.com/Parquet/parquet-compatibility/ causes an exception while reading with pyarrow 22.0.0:
```python
import urllib.request
import pathlib
import pyarrow.parquet as pq
p = pathlib.Path("nation.impala.parquet")
if not p.exists():
urllib.request.urlretrieve(
"https://github.com/Parquet/parquet-compatibility/raw/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet",
p
)
pq.read_table(p)
```
which raises with
```pytb
Traceback (most recent call last):
File "/Users/toaugspurger/gh/dask/dask/bug.py", line 12, in <module>
pq.read_table(p)
File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1899, in read_table
return dataset.read(columns=columns, use_threads=use_threads,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1538, in read
table = self._dataset.to_table(
^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 589, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3969, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid number of indices: 0
```
pyarrow 21.0.0 was able to read that file.
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3510066062
Thanks for the pointer @thisisnic . It looks like those files were produced with Polars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3460415464
Ok, I looked at this.
These are extremely old Parquet files that were generated while the [Parquet format](https://github.com/apache/parquet-format/) was still being designed.
One of the columns, of length 25, has RLE_BIT_PACKED-encoded dictionary integers over a bit width of 5. The RLE_BIT_PACKED data is 17 bytes and contains a single bit-packed run with count=4, which is really 32 values. But 17 bytes (of which 1 byte header) cannot encode 32 5-bit values (only 25). So, while the RLE_BIT_PACKED data can technically encode the 25 integers, it lacks the padding implicit in the spec.
Now, we could relax this a bit in our decoder, but the question is whether more recent files share this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3527034664
Issue resolved by pull request 47992
https://github.com/apache/arrow/pull/47992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3520614799
Thanks for the heads up @pryvkin10x . This is good to know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #47981: [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file
URL: https://github.com/apache/arrow/issues/47981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3497901741
The linked R issue here looks like it's happening with (AFAIK) more recent files: #48066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pryvkin10x (via GitHub)" <gi...@apache.org>.
pryvkin10x commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3513888876
We're seeing a different error message, in addition to this one, on similar files that were also generated with that version of polars. Can confirm that they also have this truncated RLE issue.
Is this error message caused by the same underlying issue?
`OSError: Unexpected end of stream`
https://github.com/scverse/spatialdata-io/issues/334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "pryvkin10x (via GitHub)" <gi...@apache.org>.
pryvkin10x commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3518926887
Can confirm that pitrou's commit above fixes the `Unexpected end of stream` error as well. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3457658444
I've been able to bisect and find the first commit that produces the failure for the above parquet file.
The commit: https://github.com/apache/arrow/commit/33d1f32136f950520dcca278b77acff430dc8dca fails reading the file.
The previous commit: https://github.com/apache/arrow/commit/353a0a228b94d634c309ebfd9581d08e8a5f08e4 does not fail reading the file.
This was fixing some undefined behavior found on fuzzing, we should investigate further the file to understand what is the problem but this will give us a hint. It might probably tell us that there's a problem with the file generation.
CC @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file [arrow]
Posted by "TomAugspurger (via GitHub)" <gi...@apache.org>.
TomAugspurger commented on issue #47981:
URL: https://github.com/apache/arrow/issues/47981#issuecomment-3472863230
FWIW, this just came up in one of Dask's unit tests. I suspect it's not critical, especially if this file is arguably invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org