You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yangshangqing95 (via GitHub)" <gi...@apache.org> on 2026/08/14 20:43:34 UTC
[PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
yangshangqing95 opened a new pull request, #17663:
URL: https://github.com/apache/iceberg/pull/17663
## Summary
This is the first part of #17596.
Update the vectorized Parquet position reader to use row indexes exposed by `PageReadStore` when available, preserving physical row positions across non-contiguous reads and batch boundaries.
The existing sequential behavior remains unchanged when row indexes are not available.
This also benefits vectorized fallback row ID generation, which reuses the position reader.
Related PR:
Related POC: #17597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#discussion_r3798411623
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java:
##########
@@ -694,8 +696,17 @@ public VectorHolder read(VectorHolder reuse, int numValsToRead) {
}
ArrowBuf dataBuffer = vec.getDataBuffer();
- for (int i = 0; i < numValsToRead; i += 1) {
- dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+
+ if (rowIndexes != null) {
Review Comment:
That sounds reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#issuecomment-5975253305
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#discussion_r3798123231
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java:
##########
@@ -694,8 +696,17 @@ public VectorHolder read(VectorHolder reuse, int numValsToRead) {
}
ArrowBuf dataBuffer = vec.getDataBuffer();
- for (int i = 0; i < numValsToRead; i += 1) {
- dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+
+ if (rowIndexes != null) {
Review Comment:
My thinking is that, logically, the preferred approach would be to get the sequence number from the row indexes rather than incrementing it manually. So, from a code-flow perspective, I’d lean toward putting the != null check first. That also feels more intuitive to me, but I’d be very interested to hear your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#issuecomment-5416190987
Hi @szehon-ho could you please help review this PR? Subsequent work depends on this, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#discussion_r3798075366
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java:
##########
@@ -694,8 +696,17 @@ public VectorHolder read(VectorHolder reuse, int numValsToRead) {
}
ArrowBuf dataBuffer = vec.getDataBuffer();
- for (int i = 0; i < numValsToRead; i += 1) {
- dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+
+ if (rowIndexes != null) {
Review Comment:
Hi, to maintain code style and consistency with #17662, would you mind if we keep this one as is for now? Once both PRs are merged, I can submit a separate PR to clean up and refine the code style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#issuecomment-5318979698
> Nicely done. Thank you!
>
> BTW the non-vectorized read path has the same issue. Will there be a followup to fix it?
Hi @anoopj thanks for the reviewing!
Yes the non-vectorized related PR has been approved. #17662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#discussion_r3798056434
##########
arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestVectorizedArrowReader.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.arrow.vectorized;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Optional;
+import java.util.PrimitiveIterator;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.junit.jupiter.api.Test;
+
+public class TestVectorizedArrowReader {
+
+ @Test
+ public void testPositionReaderWithSequentialRowsAcrossBatches() {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#discussion_r3797636109
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java:
##########
@@ -694,8 +696,17 @@ public VectorHolder read(VectorHolder reuse, int numValsToRead) {
}
ArrowBuf dataBuffer = vec.getDataBuffer();
- for (int i = 0; i < numValsToRead; i += 1) {
- dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+
+ if (rowIndexes != null) {
Review Comment:
Minor readability thing. Consider inverting it such that `if` covers the `null` scenario (normal contiguous read) and the `else` part handles the negation.
##########
arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestVectorizedArrowReader.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.arrow.vectorized;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Optional;
+import java.util.PrimitiveIterator;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.junit.jupiter.api.Test;
+
+public class TestVectorizedArrowReader {
+
+ @Test
+ public void testPositionReaderWithSequentialRowsAcrossBatches() {
Review Comment:
Minor: we have been trying to avoid the `test` prefix for new tests since that is no longer necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17663:
URL: https://github.com/apache/iceberg/pull/17663#issuecomment-5851283623
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Preserve non-contiguous row positions in vectorized reads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17663: Parquet: Preserve non-contiguous row positions in vectorized reads
URL: https://github.com/apache/iceberg/pull/17663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org