You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "josephperez3 (via GitHub)" <gi...@apache.org> on 2026/09/30 14:39:02 UTC
[PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
josephperez3 opened a new pull request, #15747:
URL: https://github.com/apache/iceberg/pull/15747
Closes #9497
## Problem
PR #3774 added `visitTwoLevelList()` to correctly detect 2-level lists during schema traversal, but the `list()` methods in the reader builders were not updated. `visitTwoLevelList()` does not push the repeated field name onto the `fieldNames` stack, so `currentPath()` is one segment short. This produces `defLevel=0` and `repLevel=-1` instead of the correct values, causing `RepeatedReader` to loop indefinitely and crash with `ParquetDecodingException: Reading past RLE/BitPacking stream`.
## Fix
In each `list()` method, detect 2-level lists via `isOldListElementType()` and compute def/rep levels from the element path instead of `currentPath()`. We can skip the `OptionReader` wrapping since elements are always non-null by nature of two level encoding.
Fixed for both Flink and Spark read paths.
## Testing
New tests in `TestSparkParquetReader` and `TestFlinkParquetReader`: write a 2-level Parquet file which includes some empty lists and verifies each row.
Existing tests all pass, and without the fix, the new tests crash:
```
ParquetDecodingException: Can't read value in column [names, names_tuple]
repeated binary names_tuple (STRING) at value 7 out of 7 in current page.
repetition level: 0, definition level: 2
Caused by: could not read bytes at offset 40
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "josephperez3 (via GitHub)" <gi...@apache.org>.
josephperez3 commented on code in PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#discussion_r2983888999
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetReaders.java:
##########
@@ -152,8 +152,13 @@ public ParquetValueReader<?> list(
Type elementType = ParquetSchemaUtil.determineListElementType(array);
int elementD = type.getMaxDefinitionLevel(path(elementType.getName())) - 1;
- return new ArrayReader<>(
- repeatedD, repeatedR, ParquetValueReaders.option(elementType, elementD, elementReader));
+ if (ParquetSchemaUtil.isOldListElementType(array)) {
Review Comment:
Yup that's cleaner thanks. Just pushed a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15747: Parquet: Fix readers crashing on 2-level (Thrift) list encoding
URL: https://github.com/apache/iceberg/pull/15747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "josephperez3 (via GitHub) via issues" <is...@iceberg.apache.org>.
josephperez3 commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-6750600951
Hey @manuzhang did you want to take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-4348599234
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "josephperez3 (via GitHub)" <gi...@apache.org>.
josephperez3 commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-4157073864
Hey @pvary did you get a chance to look at the updated code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "josephperez3 (via GitHub)" <gi...@apache.org>.
josephperez3 commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-6004660364
> @josephperez3 Could you please revive this PR?
Hey thanks, yup just rebased. Built a jar to verify and looks good, also added some more unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-4402221922
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#discussion_r2981381548
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetReaders.java:
##########
@@ -152,8 +152,13 @@ public ParquetValueReader<?> list(
Type elementType = ParquetSchemaUtil.determineListElementType(array);
int elementD = type.getMaxDefinitionLevel(path(elementType.getName())) - 1;
- return new ArrayReader<>(
- repeatedD, repeatedR, ParquetValueReaders.option(elementType, elementD, elementReader));
+ if (ParquetSchemaUtil.isOldListElementType(array)) {
Review Comment:
What about having this duplicated code in a single place, like:
```
public static <E> ResolvedList<E> resolveList(
MessageType fileType,
GroupType array,
String[] repeatedPath,
String[] elementPath,
ParquetValueReader<E> elementReader) {
Type elementType = ParquetSchemaUtil.determineListElementType(array);
int elementD = fileType.getMaxDefinitionLevel(elementPath) - 1;
if (ParquetSchemaUtil.isOldListElementType(array)) {
int elementR = fileType.getMaxRepetitionLevel(elementPath) - 1;
return new ResolvedList<>(elementD, elementR, elementReader);
} else {
int repeatedD = fileType.getMaxDefinitionLevel(repeatedPath) - 1;
int repeatedR = fileType.getMaxRepetitionLevel(repeatedPath) - 1;
return new ResolvedList<>(
repeatedD, repeatedR, option(elementType, elementD, elementReader));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-5913530549
@josephperez3 Could you please revive this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Fix readers crashing on 2-level (Thrift) list encoding [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #15747:
URL: https://github.com/apache/iceberg/pull/15747#issuecomment-6645924205
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org