You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jianguotian (via GitHub)" <gi...@apache.org> on 2026/09/09 05:48:38 UTC
[PR] [common] Preserve specialized iterators for full identity mappings [paimon]
jianguotian opened a new pull request, #9702:
URL: https://github.com/apache/paimon/pull/9702
### Purpose
Preserve specialized `ColumnarRowIterator` implementations when an index mapping
does not change the batch.
The read path can supply an explicit full identity mapping even when all columns
remain in their original order. `ColumnarRowIterator.mapping` currently calls
`copy` for any non-null index mapping. The base implementation of `copy` creates a
plain `ColumnarRowIterator`, so subclasses that inherit it lose their additional
batch capabilities despite the mapping being a no-op.
This change:
- Returns the original iterator when no partition mapping is present and the
index mapping is exactly `[0, 1, ..., batchArity - 1]`.
- Keeps the existing copy path for projections, reordering, missing columns, and
partition injection. A shorter identity prefix is still a projection.
- Adds regression tests for specialized-iterator preservation and the unchanged
projection, reordering, and partition-mapping behavior.
The change is confined to `paimon-common`. It does not change public API or add a
format-specific writer or compaction bridge.
### Tests
Validated commit: `d53ff92c8db26f16ebd1dffc88520e23a528aeb5`.
- `ColumnarRowIteratorTest`: **4/4 passed on JDK 8 and JDK 11**, with no skipped
tests.
- Regression proof: `testIdentityMappingPreservesSpecializedIterator` fails on
the parent production implementation because the specialized iterator is
replaced by a base iterator; the other three tests pass. All four pass with
this change.
- Non-fast Maven tests passed Checkstyle and Spotless; `paimon-common` packaging
verification with tests skipped and `git diff --check` also passed.
```bash
# Prepare the reactor dependencies for this revision.
mvn -pl paimon-common -am -Pfast-build -DskipTests install
# Run with JAVA_HOME set to JDK 8, then JDK 11.
mvn -pl paimon-common -DwildcardSuites=none -Dtest=ColumnarRowIteratorTest test
# Packaging and configured verification checks, separately from the tests above.
mvn -pl paimon-common -DskipTests verify
git diff --check HEAD^ HEAD
```
Supplementary downstream validation used an **implementation-equivalent build,
not this PR commit**: real compaction retained the specialized iterator for all
three full-identity batches, reached batch writes without generic-vector or row
fallback, and preserved row count and content. This is functional-path evidence,
not a performance benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on PR #9702:
URL: https://github.com/apache/paimon/pull/9702#issuecomment-5630661489
Thanks for verifying. Agreed that this standalone PR does not demonstrate a benefit for an existing in-tree consumer. The intended consumer is the Mosaic direct-Arrow path, which is not upstream yet.
I’ll keep #9702 closed and carry this change together with that consumer, end-to-end regression coverage, and performance evidence rather than resubmitting it independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9702:
URL: https://github.com/apache/paimon/pull/9702#issuecomment-5629947276
Thanks for addressing the row-tracking issue. I reran the 10 iterator/reader tests with the head classes on JDK 8; they pass. However, the original capability-loss premise does not hold for the current production path: Parquet and ORC create VectorizedRowIterator, which already overrides copy(ColumnVector[]) and returns a VectorizedRowIterator, preserving VectorizedRecordIterator for Arrow conversion. That override exists on the base as well as this head. The new TestingSpecializedIterator adds no production capability, so checking its object identity does not demonstrate an affected user read.
The extra DataFileRecordReader ownership logic is needed to make the new identity shortcut safe; it does not establish an independent end-to-end benefit over the existing copy path. Closing this version because that benefit has not been demonstrated and the stated existing-reader defect is already handled. We can revisit a focused change with a real affected iterator/caller and failing read, or a representative end-to-end allocation/throughput comparison showing that avoiding the copy is worthwhile. Please keep that concrete consumer in the regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian closed pull request #9702: [common][core] Preserve identity-mapped iterators safely
URL: https://github.com/apache/paimon/pull/9702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on code in PR #9702:
URL: https://github.com/apache/paimon/pull/9702#discussion_r3985693023
##########
paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRowIterator.java:
##########
@@ -115,6 +115,10 @@ public ColumnarRowIterator copy(ColumnVector[] vectors) {
public ColumnarRowIterator mapping(
@Nullable PartitionInfo partitionInfo, @Nullable int[] indexMapping) {
+ if (partitionInfo == null && isIdentityMapping(indexMapping, row.batch().getArity())) {
+ return this;
Review Comment:
Fixed. `DataFileRecordReader` now isolates the reader-owned column array before installing row-tracking vectors on an identity-mapped reusable batch, while preserving the iterator subtype and recycler. Added repeated-batch regression coverage; targeted tests pass on JDK 8 and JDK 11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #9702: [common][core] Preserve identity-mapped iterators safely
URL: https://github.com/apache/paimon/pull/9702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Preserve specialized iterators for full identity mappings [paimon]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian closed pull request #9702: [common] Preserve specialized iterators for full identity mappings
URL: https://github.com/apache/paimon/pull/9702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Preserve specialized iterators for full identity mappings [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9702:
URL: https://github.com/apache/paimon/pull/9702#discussion_r3968142704
##########
paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRowIterator.java:
##########
@@ -115,6 +115,10 @@ public ColumnarRowIterator copy(ColumnVector[] vectors) {
public ColumnarRowIterator mapping(
@Nullable PartitionInfo partitionInfo, @Nullable int[] indexMapping) {
+ if (partitionInfo == null && isIdentityMapping(indexMapping, row.batch().getArity())) {
+ return this;
Review Comment:
**[P1] Keep row-tracking wrappers out of the reader's reusable batch**
For an unpartitioned table with row tracking enabled, reading `t$row_tracking` can reach this branch with a full identity mapping: `FormatReaderMapping.Builder.trimKeyFields()` returns an explicit identity array even when the schema mapping is null. `DataFileRecordReader` then calls `assignRowTracking()`, which replaces entries in `batch.columns` in place. Returning the original iterator exposes the Parquet/ORC reader's reusable column array to those mutations, so each reused batch wraps the previous batch's wrappers. Since their `isNullAt()` always returns false, each metadata `getLong()` recursively traverses the accumulated wrappers, causing progressively slower reads and eventually `StackOverflowError`. Previously, `createMappedVectors()` plus `copy()` isolated these mutations in a separate column array.
I reproduced this on JDK 8 with a real Parquet file, batch size 1, and the same `mapping(...).assignRowTracking(...)` sequence: this implementation overflows when checking the row at approximately 20,000 batches, while the identical test with the parent implementation completes all 100,000 batches. The four existing PR tests pass but do not exercise this reuse path.
Please keep row-tracking decoration isolated from the format reader's column array, or retain the copy path when row tracking needs to modify the vectors, and add a regression covering repeated batch reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org