You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/09/19 16:27:07 UTC
[I] Parquet read fails when written by parquet-go [parquet-java]
ArnavBalyan opened a new issue, #3320:
URL: https://github.com/apache/parquet-java/issues/3320
### Describe the enhancement requested
Parquet-go can write extra entries in the column index for 1 column chunk. This causes Parquet java to fail with java.lang.ArrayIndexOutOfBoundsException when using column indexes.
- Parquet java expects the same number of entries in column index and offset index (one entry per data page).
- When they don't match, the reader seems to be trying to read a non existent page (based on the column stats) and fails. (stacktrace below)
Structure of indexes which lead to the failure:
DEBUG: COLUMN1 ColumnIndex=5 stats
Entry 0: nullPage=false, nullCount=0, min='', max=''
Entry 1: nullPage=false, nullCount=0, min='xyz', max='abc'
Entry 2: nullPage=false, nullCount=0, min='xyz', max='abc'
Entry 3: nullPage=false, nullCount=0, min='xyz', max='abc'
Entry 4: nullPage=false, nullCount=0, min='xyz', max='abc'
DEBUG: COLUMN1 OffsetIndex=4 pages
Entry 0: offset=..., compressedSize=..., firstRowIndex=0
Entry 1: offset=..., compressedSize=..., firstRowIndex=30
Entry 2: offset=..., compressedSize=..., firstRowIndex=60
Entry 3: offset=..., compressedSize=..., firstRowIndex=90
(Here we have 1 additional column index for the same data page)
There could be potentially 2 ways to fix this:
1. Detect inconsistent metadata and do not rely on column index for unsuitable/incomplete metadata. Log a WARN and continue reading the file without index, which will correctly read the file.
2. Clearly throw an exception indicating the missing/unhandled metadata (in this case, the mismatch in number of column/offset index), so user can disable column index or correct upstream.
Stacktrace:
```
at org.apache.parquet.internal.column.columnindex.OffsetIndexBuilder$OffsetIndexImpl.getFirstRowIndex(OffsetIndexBuilder.java:66)
at org.apache.parquet.internal.filter2.columnindex.RowRanges.create(RowRanges.java:144)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.applyPredicate(ColumnIndexFilter.java:189)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.visit(ColumnIndexFilter.java:126)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.visit(ColumnIndexFilter.java:57)
at org.apache.parquet.filter2.predicate.Operators$NotEq.accept(Operators.java:192)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter$1.visit(ColumnIndexFilter.java:87)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter$1.visit(ColumnIndexFilter.java:82)
at org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:149)
at org.apache.parquet.internal.filter2.columnindex.ColumnIndexFilter.calculateRowRanges(ColumnIndexFilter.java:82)
at org.apache.parquet.hadoop.ParquetFileReader.getRowRanges(ParquetFileReader.java:1219)
at org.apache.parquet.hadoop.ParquetFileReader.getFilteredRecordCount(ParquetFileReader.java:875)
...
```
### Component(s)
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3316041432
Is this a known issue of Parquet-go?
cc @zeroshade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3316759277
Looks like it's one of the several open source implementations of parquet-go. the upstream is buggy, will check thanks for taking a look @wgtmac @zeroshade. I think it may be helpful to add an improved error message when the read fails due to corrupt metadata, will try to improve it thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #3320: Parquet read fails in certain scenarios for files written by parquet-go
URL: https://github.com/apache/parquet-java/issues/3320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3317170187
Related to handling such issues in parquet-java during reading, I would vote on the 1st approach. Statistics are not required to access the data so we should not fail but ignore invalid statistics. We shall log a proper message to make the situation clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3316066920
This is not a known issue that I'm aware of. Importantly, is this github.com/parquet-go/parquet-go? Or is this the Go library for parquet in the arrow-go repo?
If it's the latter, can you share the code that was used so I can fix this in the Go library? If it's the former, we should file an issue on parquet-go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3312865144
cc @gszadovszky just wanted to know your thoughts. Ignoring column index and reading seems to avoid the issue but may silently introduce a dependency on incorrect behaviour from upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Parquet read fails in certain scenarios for files written by parquet-go [parquet-java]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on issue #3320:
URL: https://github.com/apache/parquet-java/issues/3320#issuecomment-3318419912
Yes agreed, let me see if there is a clean way to do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org