You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "skyon0522-ai (via GitHub)" <gi...@apache.org> on 2026/10/18 14:08:11 UTC
[PR] GH-3043: Ignore V1 level encodings for omitted streams [parquet-java]
skyon0522-ai opened a new pull request, #3867:
URL: https://github.com/apache/parquet-java/pull/3867
Closes #3043.
### Rationale for this change
The attached nanoparquet file uses PLAIN in the V1 repetition/definition
encoding fields for flat required columns. Neither level stream is present.
`readPageV1` nevertheless selects a reader from each header encoding, which
can consume value bytes or fail at `ValuesReader.readInteger` before the
first record is returned. The [format specification](https://github.com/apache/parquet-format/blob/53b8bd8576827080d4382fa457003f45b41cda6e/README.md#L193-L200)
specifies that these zero-maximum level streams are omitted.
### What changes are included in this PR?
Select the existing `ZeroIntegerValuesReader` independently when the
corresponding maximum level is zero. Keep the existing encoding dispatch
for nonzero levels. The regression uses the upstream V1 writer to generate
two pages, replaces only unused level-encoding fields, and checks exact
values and levels for required, optional and repeated columns.
### Are these changes tested?
Validation on Linux (Ubuntu 24.04, Java 17.0.20.1, Maven 3.9.11):
- The new parameterized cases produced four errors on unchanged production
code. The two controls and seven existing cases passed. All 13 passed
with the fix.
- The full `parquet-column` reactor test run passed all 1,094 tests.
- A separate 92-test Hadoop run passed, including reading all 32 rows of
the issue attachment through both reader APIs and a generated control.
- Spotless checks for the two changed files, normal RAT checks across
eight reactor modules, and `git diff --check` passed.
The entire root reactor, native Windows and macOS were not tested.
### Are there any user-facing changes?
Files such as the issue attachment become readable when a V1 header uses
PLAIN for an omitted level stream.
OpenAI Codex assisted with the implementation and tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3043: Ignore V1 level encodings for omitted streams [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on PR #3867:
URL: https://github.com/apache/parquet-java/pull/3867#issuecomment-6445182346
Here's how I understand the issue (full disclosure: arrived at with the help of a conversation with codex, but these words are mine). The v1 specification [requires](https://github.com/apache/parquet-java/blob/master/parquet-format-structures/src/main/thrift/parquet.thrift#L712-L716) `definition_level_encoding` and `repetition_level_encoding` but sometimes they aren't actually relevant. This change first looks to the schema to determine if they're relevant and *then* uses them as opposed to using them by default, which can lead to issues like this.
Based on that understanding I am +1 (non-binding).
---
Here's a minimum reproduction of the issue furnished by codex.
```java
@Test
public void requiredDoubleIgnoresAbsentRepetitionLevelEncoding() throws Exception {
ColumnDescriptor column = MessageTypeParser
.parseMessageType("message test { required double foo; }")
.getColumns()
.get(0);
byte[] data = ByteBuffer.allocate(Double.BYTES)
.order(ByteOrder.LITTLE_ENDIAN)
.putDouble(12.5)
.array();
DataPageV1 page = new DataPageV1(
BytesInput.from(data),
1,
data.length,
new DoubleStatistics(),
Encoding.PLAIN, // repetition levels: absent despite this label
Encoding.RLE, // definition levels: also absent
Encoding.PLAIN);
ColumnReader reader = new ColumnReaderImpl(
column,
new MemPageReader(1, List.<DataPage>of(page).iterator(), null),
new PrimitiveConverter() {},
VersionParser.parse(Version.FULL_VERSION));
assertThat(reader.getDouble()).isEqualTo(12.5);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3043: Ignore V1 level encodings for omitted streams [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3867:
URL: https://github.com/apache/parquet-java/pull/3867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org