You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jianguotian (via GitHub)" <gi...@apache.org> on 2026/09/09 05:48:38 UTC

[PR] [common] Preserve specialized iterators for full identity mappings [paimon]

jianguotian opened a new pull request, #9702:
URL: https://github.com/apache/paimon/pull/9702

   ### Purpose
   
   Preserve specialized `ColumnarRowIterator` implementations when an index mapping
   does not change the batch.
   
   The read path can supply an explicit full identity mapping even when all columns
   remain in their original order. `ColumnarRowIterator.mapping` currently calls
   `copy` for any non-null index mapping. The base implementation of `copy` creates a
   plain `ColumnarRowIterator`, so subclasses that inherit it lose their additional
   batch capabilities despite the mapping being a no-op.
   
   This change:
   
   - Returns the original iterator when no partition mapping is present and the
     index mapping is exactly `[0, 1, ..., batchArity - 1]`.
   - Keeps the existing copy path for projections, reordering, missing columns, and
     partition injection. A shorter identity prefix is still a projection.
   - Adds regression tests for specialized-iterator preservation and the unchanged
     projection, reordering, and partition-mapping behavior.
   
   The change is confined to `paimon-common`. It does not change public API or add a
   format-specific writer or compaction bridge.
   
   ### Tests
   
   Validated commit: `d53ff92c8db26f16ebd1dffc88520e23a528aeb5`.
   
   - `ColumnarRowIteratorTest`: **4/4 passed on JDK 8 and JDK 11**, with no skipped
     tests.
   - Regression proof: `testIdentityMappingPreservesSpecializedIterator` fails on
     the parent production implementation because the specialized iterator is
     replaced by a base iterator; the other three tests pass. All four pass with
     this change.
   - Non-fast Maven tests passed Checkstyle and Spotless; `paimon-common` packaging
     verification with tests skipped and `git diff --check` also passed.
   
   ```bash
   # Prepare the reactor dependencies for this revision.
   mvn -pl paimon-common -am -Pfast-build -DskipTests install
   
   # Run with JAVA_HOME set to JDK 8, then JDK 11.
   mvn -pl paimon-common -DwildcardSuites=none -Dtest=ColumnarRowIteratorTest test
   
   # Packaging and configured verification checks, separately from the tests above.
   mvn -pl paimon-common -DskipTests verify
   git diff --check HEAD^ HEAD
   ```
   
   Supplementary downstream validation used an **implementation-equivalent build,
   not this PR commit**: real compaction retained the specialized iterator for all
   three full-identity batches, reached batch writes without generic-vector or row
   fallback, and preserved row count and content. This is functional-path evidence,
   not a performance benchmark.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]

Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on PR #9702:
URL: https://github.com/apache/paimon/pull/9702#issuecomment-5630661489

   Thanks for verifying. Agreed that this standalone PR does not demonstrate a benefit for an existing in-tree consumer. The intended consumer is the Mosaic direct-Arrow path, which is not upstream yet.
   
   I’ll keep #9702 closed and carry this change together with that consumer, end-to-end regression coverage, and performance evidence rather than resubmitting it independently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9702:
URL: https://github.com/apache/paimon/pull/9702#issuecomment-5629947276

   Thanks for addressing the row-tracking issue. I reran the 10 iterator/reader tests with the head classes on JDK 8; they pass. However, the original capability-loss premise does not hold for the current production path: Parquet and ORC create VectorizedRowIterator, which already overrides copy(ColumnVector[]) and returns a VectorizedRowIterator, preserving VectorizedRecordIterator for Arrow conversion. That override exists on the base as well as this head. The new TestingSpecializedIterator adds no production capability, so checking its object identity does not demonstrate an affected user read.
   
   The extra DataFileRecordReader ownership logic is needed to make the new identity shortcut safe; it does not establish an independent end-to-end benefit over the existing copy path. Closing this version because that benefit has not been demonstrated and the stated existing-reader defect is already handled. We can revisit a focused change with a real affected iterator/caller and failing read, or a representative end-to-end allocation/throughput comparison showing that avoiding the copy is worthwhile. Please keep that concrete consumer in the regression test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]

Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian closed pull request #9702: [common][core] Preserve identity-mapped iterators safely
URL: https://github.com/apache/paimon/pull/9702


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]

Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on code in PR #9702:
URL: https://github.com/apache/paimon/pull/9702#discussion_r3985693023


##########
paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRowIterator.java:
##########
@@ -115,6 +115,10 @@ public ColumnarRowIterator copy(ColumnVector[] vectors) {
 
     public ColumnarRowIterator mapping(
             @Nullable PartitionInfo partitionInfo, @Nullable int[] indexMapping) {
+        if (partitionInfo == null && isIdentityMapping(indexMapping, row.batch().getArity())) {
+            return this;

Review Comment:
   Fixed. `DataFileRecordReader` now isolates the reader-owned column array before installing row-tracking vectors on an identity-mapped reusable batch, while preserving the iterator subtype and recycler. Added repeated-batch regression coverage; targeted tests pass on JDK 8 and JDK 11.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common][core] Preserve identity-mapped iterators safely [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #9702: [common][core] Preserve identity-mapped iterators safely
URL: https://github.com/apache/paimon/pull/9702


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Preserve specialized iterators for full identity mappings [paimon]

Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian closed pull request #9702: [common] Preserve specialized iterators for full identity mappings
URL: https://github.com/apache/paimon/pull/9702


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Preserve specialized iterators for full identity mappings [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9702:
URL: https://github.com/apache/paimon/pull/9702#discussion_r3968142704


##########
paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRowIterator.java:
##########
@@ -115,6 +115,10 @@ public ColumnarRowIterator copy(ColumnVector[] vectors) {
 
     public ColumnarRowIterator mapping(
             @Nullable PartitionInfo partitionInfo, @Nullable int[] indexMapping) {
+        if (partitionInfo == null && isIdentityMapping(indexMapping, row.batch().getArity())) {
+            return this;

Review Comment:
   **[P1] Keep row-tracking wrappers out of the reader's reusable batch**
   
   For an unpartitioned table with row tracking enabled, reading `t$row_tracking` can reach this branch with a full identity mapping: `FormatReaderMapping.Builder.trimKeyFields()` returns an explicit identity array even when the schema mapping is null. `DataFileRecordReader` then calls `assignRowTracking()`, which replaces entries in `batch.columns` in place. Returning the original iterator exposes the Parquet/ORC reader's reusable column array to those mutations, so each reused batch wraps the previous batch's wrappers. Since their `isNullAt()` always returns false, each metadata `getLong()` recursively traverses the accumulated wrappers, causing progressively slower reads and eventually `StackOverflowError`. Previously, `createMappedVectors()` plus `copy()` isolated these mutations in a separate column array.
   
   I reproduced this on JDK 8 with a real Parquet file, batch size 1, and the same `mapping(...).assignRowTracking(...)` sequence: this implementation overflows when checking the row at approximately 20,000 batches, while the identical test with the parent implementation completes all 100,000 batches. The four existing PR tests pass but do not exercise this reuse path.
   
   Please keep row-tracking decoration isolated from the format reader's column array, or retain the copy path when row tracking needs to modify the vectors, and add a regression covering repeated batch reuse.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org