You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/11 18:13:05 UTC

[GitHub] [iceberg] aokolnychyi opened a new pull request #3533: Arrow: Fix vectorized position reader

aokolnychyi opened a new pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533


   This PR fixes the vectorized position reader. Previously, it would give wrong results if multiple batches were read.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-967275681


   @chenjunjiedada, our Spark code always reuses containers. The test I added for multiple batches triggers it and failed before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748448887



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       We can make the same assumption as in `setNull` (i.e. the set indices don't overlap) and then simply add the number of set nulls. I'd probably just call `numNulls += num`.
   
   Thoughts, @samarthjain @RussellSpitzer @rdblue?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-969433231


   Thanks for reviewing, @flyrain @rdblue @RussellSpitzer @chenjunjiedada @samarthjain @nastra @kbendick !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749641779



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);
+      } else {
         vec.setValueCount(0);
         nulls.reset();
-      } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      ArrowBuf validityBuffer = vec.getValidityBuffer();
+
+      for (int i = 0; i < numValsToRead; i += 1) {

Review comment:
       We use exactly the same pattern in a few other places. I kind of hoped the complier would be smart enough to rewrite this efficiently as the condition is static but I agree with you. I'll update.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749687399



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +469,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+    }
+
+    @Override
+    public void close() {

Review comment:
       Yeah, looks like `ColumnarBatchReader` does close each vector too. I'll change it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747739317



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       Again I think the assumption of this class is that you never call a "set" method more than once on an index and that you call it on every index.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?
     System.arraycopy(nulls, 0, isNull, startIndex, num);
   }
 
   public void setNotNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Unless we were unsetting a value that was previously set as null I don't think we would change things here. This api basically requires you to only call "set" on an index a single time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748885547



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      ArrowBuf validityBuffer = vec.getValidityBuffer();
+
+      for (int i = 0; i < numValsToRead; i += 1) {
+        dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+        if (setArrowValidityVector) {
+          BitVectorHelper.setBit(validityBuffer, i);
         }
-        nulls = new NullabilityHolder(numValsToRead);
       }
 
       rowStart += numValsToRead;
       vec.setValueCount(numValsToRead);
-      nulls.setNotNulls(0, numValsToRead);
 
       return new VectorHolder.PositionVectorHolder(vec, MetadataColumns.ROW_POSITION.type(), nulls);
     }
 
+    private BigIntVector newVector() {

Review comment:
       Nit: `newVector` and `newNullabilityHolder` can be static by passing in the allocator and batch size. I'd also make `arrowField` static because it is a field for a static column definition, `MetadataColumns.ROW_POSITION`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748885681



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +473,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;

Review comment:
       I think this needs to reallocate the nullability holder if batch size increased, or else a batch size change could cause it an underflow when accessing nullability. Right now, it doesn't really matter because batch size never changes (it's set during the vectorized parquet constructor) but this should still implement the contract, I think.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748446761



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       I think it is fine the way it is. We have to set the value for each index. Otherwise, no guarantee it is going to be correct.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748886314



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);
+      } else {
         vec.setValueCount(0);
         nulls.reset();
-      } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      ArrowBuf validityBuffer = vec.getValidityBuffer();
+
+      for (int i = 0; i < numValsToRead; i += 1) {

Review comment:
       I think this should move back to two loops. The point of two is to avoid `if` statements in a loop and to maximize the chances that the memory regions that are used are in cache. High-level control flow and narrow memory access are how we get the benefits of vectorization.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748885681



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +473,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;

Review comment:
       I think this needs to reallocate the nullability holder if batch size increased, or else a batch size change could cause it to overflow.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer edited a comment on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966551983


   > @aokolnychyi, what is the problem that you're fixing here?
   
   We hit a bug when reading the position column over batch boundaries. In general this manifests if you attempt to read the column in any but the first batch where you get a null where a Long should be. 
   ```
   Caused by: java.lang.IllegalStateException: Value at index is null
     at org.apache.iceberg.shaded.org.apache.arrow.vector.BigIntVector.get(BigIntVector.java:112)
   ```
   
   This has downstream effects of any vectorized read which uses the result of the _pos column (like applying deletes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966551281


   @aokolnychyi, what is the problem that you're fixing here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747727287



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java
##########
@@ -52,14 +59,17 @@
 import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
 import static org.apache.iceberg.TableProperties.FORMAT_VERSION;
 import static org.apache.iceberg.TableProperties.ORC_VECTORIZATION_ENABLED;
+import static org.apache.iceberg.TableProperties.PARQUET_BATCH_SIZE;
+import static org.apache.iceberg.TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES;
 import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
+import static org.apache.spark.sql.functions.lit;
 
 @RunWith(Parameterized.class)
 public class TestSparkMetadataColumns extends SparkTestBase {
 
   private static final String TABLE_NAME = "test_table";
   private static final Schema SCHEMA = new Schema(
-      Types.NestedField.required(1, "id", Types.LongType.get()),

Review comment:
       Not sure I understand this mod




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749653784



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       While I agree with how the reuse logic should behave theoretically, it is not we have in `VectorizedArrowReader`. I looked at that part as an example of how it should behave. I am a bit worried about unforeseen consequences of changing just this place. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749677851



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +469,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+    }
+
+    @Override
+    public void close() {

Review comment:
       We should probably remove that implementation of `close`. The vectorized buffers live outside of the readers. Close should be used to close file resources, not data in memory resources.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749713223



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +470,19 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+      this.nulls = newNullabilityHolder(batchSize);

Review comment:
       Shouldn't we only do this if the batch size changed and not every time this is called?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747736074



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       It definitely seems like we should be adding to numNulls here. The only consumers of this I see are in the classes Extending the Spark ColumnVector class which does expect this value to be correct.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966526685


   @samarthjain @rdblue @chenjunjiedada @rymurr @nastra @RussellSpitzer @flyrain @szehon-ho @karuppayya


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748886830



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +469,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+    }
+
+    @Override
+    public void close() {

Review comment:
       I don't think that it is the responsibility of the reader to close vectors, since vectors are not owned by readers.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748885681



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +473,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;

Review comment:
       I think this needs to reallocate the nullability holder if batch size increased, or else a batch size change could cause it an underflow when accessing nullability.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748886772



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       There is no guarantee that the instance variable `vec` is the same one that backs the reused `VectorHolder`. The method contract is that this will fill the reused vector with data, not that it will reuse a vector if the holder is non-null. I think that the `else` case here should set the _local_ variable `vec` to `reuse.vector()`. Technically, we should probably use the same nullability holder as well, but I think that it is okay to return a new `VectorHolder` with the constant nullability holder instead.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749712875



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,52 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private static final Field ROW_POSITION_ARROW_FIELD = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      FieldVector vec;
+      if (reuse == null) {
+        vec = newVector(batchSize);
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
+        FieldVector reusedVector = reuse.vector();
+        reusedVector.setValueCount(0);
+        vec = reusedVector;
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      for (int i = 0; i < numValsToRead; i += 1) {
+        dataBuffer.setLong((long) i * Long.BYTES, rowStart + i);
+      }
+
+      if (setArrowValidityVector) {
+        ArrowBuf validityBuffer = vec.getValidityBuffer();
         for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+          BitVectorHelper.setBit(validityBuffer, i);
         }
-        nulls = new NullabilityHolder(numValsToRead);
       }
 
       rowStart += numValsToRead;
       vec.setValueCount(numValsToRead);
-      nulls.setNotNulls(0, numValsToRead);
 
       return new VectorHolder.PositionVectorHolder(vec, MetadataColumns.ROW_POSITION.type(), nulls);
     }
 
+    private BigIntVector newVector(int valueCount) {

Review comment:
       Could be `static` if we pass in the allocator as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747791488



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       You are, probably, right. Looks like the assumption is that the caller would have to call `setXXX` for each row in the batch after `reset`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer edited a comment on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966551983


   > @aokolnychyi, what is the problem that you're fixing here?
   
   We hit a bug when reading the position column over batch boundaries. In general this manifests if you attempt to read the column as getting a null where a Long should be. 
   ```
   Caused by: java.lang.IllegalStateException: Value at index is null
     at org.apache.iceberg.shaded.org.apache.arrow.vector.BigIntVector.get(BigIntVector.java:112)
   ```
   
   This has downstream effects of any vectorized read which uses the result of the _pos column (like applying deletes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi edited a comment on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966592919


   Here is the old logic that I think was causing the issue:
   
   ```
   if (reuse != null) {
     vec.setValueCount(0);
     nulls.reset();
   } else {
     ...
   }
   ```
   
   We never populated values beyond the first batch when container reuse was enabled (always?).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747731130



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java
##########
@@ -52,14 +59,17 @@
 import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
 import static org.apache.iceberg.TableProperties.FORMAT_VERSION;
 import static org.apache.iceberg.TableProperties.ORC_VECTORIZATION_ENABLED;
+import static org.apache.iceberg.TableProperties.PARQUET_BATCH_SIZE;
+import static org.apache.iceberg.TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES;
 import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
+import static org.apache.spark.sql.functions.lit;
 
 @RunWith(Parameterized.class)
 public class TestSparkMetadataColumns extends SparkTestBase {
 
   private static final String TABLE_NAME = "test_table";
   private static final Schema SCHEMA = new Schema(
-      Types.NestedField.required(1, "id", Types.LongType.get()),

Review comment:
       Was hitting the cannot write nullable data to non-null column in tests.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747720844



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       There are cases when we reuse the nullability holder and call `reset` and `setNotNulls(0, numValsToRead)`. It looks like we still have to invalidate the underlying values?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747801789



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Yeah, looks like we should add to `numNulls` here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] samarthjain commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
samarthjain commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748053654



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       That's correct, @aokolnychyi . When I wrote this, there was inherent assumption that the user of this class will be calling `setNull(index)` and `setNotNull(index)` for every row in the batch which is why reset() didn't reinitialize or assign a new `byte[]` for `isNull`. See the comment at the top the class
   ```
   /**
    * Instances of this class simply track whether a value at an index is null.
    * For simplicity and performance, it is expected that various setter methods
    * {@link #setNull(int)}, {@link #setNulls(int, int)}, {@link #setNotNull(int)}
    * and {@link #setNotNulls(int, int)} are invoked with monotonically
    * increasing values for the index parameter.
    */
    ```
   Although, looking at this code now, I agree with @RussellSpitzer that we could simply reinitialize the `isNull` array again. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749712218



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);
+      } else {
         vec.setValueCount(0);
         nulls.reset();
-      } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      ArrowBuf validityBuffer = vec.getValidityBuffer();
+
+      for (int i = 0; i < numValsToRead; i += 1) {

Review comment:
       We do? We should definitely avoid branching inside these loops!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747748752



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);

Review comment:
       Yeah, why don't we always use the same one?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747746948



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       @samarthjain can you comment on this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966551983


   > @aokolnychyi, what is the problem that you're fixing here?
   
   We hit a bug when reading the position column over batch boundaries. This has downstream effects of any vectorized read which uses the result of the _pos column (like applying deletes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] chenjunjiedada commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966792576


   > Here is the old logic that I think was causing the issue:
   > 
   > ```
   > if (reuse != null) {
   >   vec.setValueCount(0);
   >   nulls.reset();
   > } else {
   >   ...
   > }
   > ```
   > 
   > We never populated values beyond the first batch when container reuse was enabled (always?).
   
   Thanks for the fix, I think that is it.  Could you please help to add a unit test that reuses the container?  Or do the unit tests in this PR already cover it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747897080



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       The challenge is also what number we should add. Are there overlaps, for example? I don't think we can answer that without iterating through the array.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] samarthjain commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
samarthjain commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748589010



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Agreed, we should be adding to `numNulls` here. Looking at the caller of `setNulls()`, it looks like we call this method in cases where RLE encoding and the definition level tells us that next num values are null. So it should be ok to do `numNulls += num`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966592919


   Here is the old logic that I think was causing the issue:
   
   ```
   if (reuse != null) {
     vec.setValueCount(0);
     nulls.reset();
   } else {
     ...
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747803033



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       I think it is the only place we need to change in this class.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748450646



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       I had that initially but then we [discussed](https://github.com/apache/iceberg/pull/3533#discussion_r747722269) that the values are never null so calling `setNotNulls` for each batch seemed redundant. That's why I moved that to the method that inits the nullability holder and simply reuse the same nullability holder. What do you think, @nastra?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747722269



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);

Review comment:
       Technically, we can always reuse the same nullability holder with no nulls as the column is not nullable. The existing code still calls `setNotNulls` for each batch.

##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java
##########
@@ -158,6 +168,62 @@ public void testSpecAndPartitionMetadataColumns() {
         sql("SELECT _spec_id, _partition FROM %s ORDER BY _spec_id", TABLE_NAME));
   }
 
+  @Test
+  public void testPositionMetadataColumnWithMultipleRowGroups() throws NoSuchTableException {
+    Assume.assumeTrue(fileFormat == FileFormat.PARQUET);
+
+    table.updateProperties()
+        .set(PARQUET_ROW_GROUP_SIZE_BYTES, "100")
+        .commit();
+
+    List<Long> ids = Lists.newArrayList();
+    for (long id = 0L; id < 200L; id++) {
+      ids.add(id);
+    }
+    Dataset<Row> df = spark.createDataset(ids, Encoders.LONG())
+        .withColumnRenamed("value", "id")
+        .withColumn("category", lit("hr"))
+        .withColumn("data", lit("ABCDEF"));
+    df.coalesce(1).writeTo(TABLE_NAME).append();
+
+    Assert.assertEquals(200, spark.table(TABLE_NAME).count());
+
+    List<Object[]> expectedRows = ids.stream()
+        .map(this::row)
+        .collect(Collectors.toList());
+    assertEquals("Rows must match",
+        expectedRows,
+        sql("SELECT _pos FROM %s", TABLE_NAME));
+  }
+
+  @Test
+  public void testPositionMetadataColumnWithMultipleBatches() throws NoSuchTableException {

Review comment:
       This test would previously fail and output wrong results. I tested it triggers the bug.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       I'd appreciate if someone could explain why we only modify `numNulls` in `setNull`.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +469,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+    }
+
+    @Override
+    public void close() {

Review comment:
       I think we did not close the vector before.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);
+      } else {
         vec.setValueCount(0);
         nulls.reset();
-      } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+      }
+
+      ArrowBuf dataBuffer = vec.getDataBuffer();
+      ArrowBuf validityBuffer = vec.getValidityBuffer();
+
+      for (int i = 0; i < numValsToRead; i += 1) {

Review comment:
       I switched to a single for loop and setting the Arrow validity vector only if required.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {

Review comment:
       We don't have to extend `VectorizedArrowReader` as this class has nothing in common but there are places in code that do a cast and it would fail otherwise.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747739317



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       Again I think the assumption of this class is that you never call a "set" method more than once on an index and that you call it on every index. A memset of the whole vector seems really cheap and safe to do though, so i wouldn't mind if we did it here as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747766166



##########
File path: spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java
##########
@@ -158,6 +168,62 @@ public void testSpecAndPartitionMetadataColumns() {
         sql("SELECT _spec_id, _partition FROM %s ORDER BY _spec_id", TABLE_NAME));
   }
 
+  @Test
+  public void testPositionMetadataColumnWithMultipleRowGroups() throws NoSuchTableException {
+    Assume.assumeTrue(fileFormat == FileFormat.PARQUET);
+
+    table.updateProperties()
+        .set(PARQUET_ROW_GROUP_SIZE_BYTES, "100")
+        .commit();
+
+    List<Long> ids = Lists.newArrayList();
+    for (long id = 0L; id < 200L; id++) {
+      ids.add(id);
+    }
+    Dataset<Row> df = spark.createDataset(ids, Encoders.LONG())
+        .withColumnRenamed("value", "id")
+        .withColumn("category", lit("hr"))
+        .withColumn("data", lit("ABCDEF"));
+    df.coalesce(1).writeTo(TABLE_NAME).append();
+
+    Assert.assertEquals(200, spark.table(TABLE_NAME).count());
+
+    List<Object[]> expectedRows = ids.stream()
+        .map(this::row)
+        .collect(Collectors.toList());
+    assertEquals("Rows must match",
+        expectedRows,
+        sql("SELECT _pos FROM %s", TABLE_NAME));
+  }
+
+  @Test
+  public void testPositionMetadataColumnWithMultipleBatches() throws NoSuchTableException {

Review comment:
       This does make me think we should add a test suite somewhere that just does "Vectorized vs NonVectorized" and just checks that with a variety of  reads/schemas




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749644011



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +469,14 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+    }
+
+    @Override
+    public void close() {

Review comment:
       I matched the logic of the main reader where we have this:
   
   ```
     @Override
     public void close() {
       if (vec != null) {
         vec.close();
       }
     }
   ```
   
   Looks like we are closing `FieldVector` for regular reads.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi merged pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi merged pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747751003



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);

Review comment:
       I was wondering why we are doing this with a nullability holder since we know it can never be nullable




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748451956



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       We currently just assume the caller is carful, I think we can just keep that assumption




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748672314



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?
     System.arraycopy(nulls, 0, isNull, startIndex, num);
   }
 
   public void setNotNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Not applies anymore.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-967767790


   Okay, I think we clarified the expected behavior of `NullabilityHolder`. I made the corresponding change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749656432



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       In `VectorizedArrowReader`, we also create a new `VectorHolder` even if the reuse is enabled.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] chenjunjiedada edited a comment on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
chenjunjiedada edited a comment on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966792576


   > Here is the old logic that I think was causing the issue:
   > 
   > ```
   > if (reuse != null) {
   >   vec.setValueCount(0);
   >   nulls.reset();
   > } else {
   >   ...
   > }
   > ```
   > 
   > We never populated values beyond the first batch when container reuse was enabled (always?).
   
   Thanks for the fix, I think that is it.  Could you please help to add a unit test that reuses the container?  Or do the unit tests in this PR already cover it since reusing is default behavior?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749712762



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -448,6 +470,19 @@ public String toString() {
 
     @Override
     public void setBatchSize(int batchSize) {
+      this.batchSize = (batchSize == 0) ? DEFAULT_BATCH_SIZE : batchSize;
+      this.nulls = newNullabilityHolder(batchSize);
+    }
+
+    private NullabilityHolder newNullabilityHolder(int size) {

Review comment:
       `static`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747793786



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);

Review comment:
       I'll probably switch to a static nullability holder with all non-nulls then.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747791737



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       It is not documented, though. @samarthjain, could you confirm our understanding?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748055193



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       do we need to call `nulls.reset();` here when reusing? Also do we need to call `nulls.setNotNulls(0, numValsToRead)` for both cases?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747791488



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       You are probably right. Looks like the assumption is that the caller would have to call `setXXX` for each row in the batch after `reset`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r749678776



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();
       } else {
-        ((BigIntVector) vec).allocateNew(numValsToRead);
-        for (int i = 0; i < numValsToRead; i += 1) {
-          vec.getDataBuffer().setLong(i * Long.BYTES, rowStart + i);
-        }
-        for (int i = 0; i < numValsToRead; i += 1) {
-          BitVectorHelper.setBit(vec.getValidityBuffer(), i);
+        vec.setValueCount(0);

Review comment:
       A new `VectorHolder` is fine and shouldn't hurt anything. We need to eventually fix the other places as well, but I don't see a reason why it would hurt to have this place implement the contract properly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748886868



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java
##########
@@ -85,7 +85,11 @@ public VectorizedReaderBuilder(
       if (idToConstant.containsKey(id)) {
         reorderedFields.add(new VectorizedArrowReader.ConstantVectorReader<>(idToConstant.get(id)));
       } else if (id == MetadataColumns.ROW_POSITION.fieldId()) {
-        reorderedFields.add(VectorizedArrowReader.positions());
+        if (setArrowValidityVector) {

Review comment:
       +1




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748886571



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);

Review comment:
       This can be static, `POS_ARROW_FIELD`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#issuecomment-966587682


   @rdblue @RussellSpitzer, it was either the exception Russell mentioned or simply wrong results. I had two identical queries with and without vectorization and they were giving different results. Based on what I see, the underlying issue is the wrong logic for handling reused batches. When container reuse was enabled and we had multiple batches, the second batch would be corrupted. I'd appreciate extra attention to that part.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748672195



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -75,6 +77,7 @@ public int numNulls() {
   }
 
   public void reset() {
+    // TODO: do we have reset the values too?

Review comment:
       I think this can be resolved now.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Fixed.

##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,26 +412,36 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = new NullabilityHolder(batchSize);

Review comment:
       Switched.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r748885961



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java
##########
@@ -407,35 +412,55 @@ public void setBatchSize(int batchSize) {
   }
 
   private static final class PositionVectorReader extends VectorizedArrowReader {
+    private final Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
+    private final BufferAllocator bufferAllocator = ArrowAllocation.rootAllocator();
+    private final boolean setArrowValidityVector;
     private long rowStart;
+    private int batchSize;
+    private FieldVector vec;
     private NullabilityHolder nulls;
 
+    PositionVectorReader(boolean setArrowValidityVector) {
+      this.setArrowValidityVector = setArrowValidityVector;
+    }
+
     @Override
     public VectorHolder read(VectorHolder reuse, int numValsToRead) {
-      Field arrowField = ArrowSchemaUtil.convert(MetadataColumns.ROW_POSITION);
-      FieldVector vec = arrowField.createVector(ArrowAllocation.rootAllocator());
-
-      if (reuse != null) {
-        vec.setValueCount(0);
-        nulls.reset();
+      if (reuse == null) {
+        this.vec = newVector();
+        this.nulls = newNullabilityHolder();

Review comment:
       I'd move the nullability holder allocation to `setBatchSize` in case the batch size changes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3533: Arrow: Fix vectorized position reader

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3533:
URL: https://github.com/apache/iceberg/pull/3533#discussion_r747802430



##########
File path: arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.java
##########
@@ -52,10 +52,12 @@ public void setNotNull(int index) {
   }
 
   public void setNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?
     System.arraycopy(nulls, 0, isNull, startIndex, num);
   }
 
   public void setNotNulls(int startIndex, int num) {
+    // TODO: do we have to modify numNulls?

Review comment:
       Yeah, agree.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org