You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/15 20:25:09 UTC
[GitHub] [iceberg] flyrain opened a new pull request #3557: Spark: Support vectorized reads with equality deletes
flyrain opened a new pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557
### Why do we need this PR?
Vectorized reads with position deletes is supported after #3287. This PR is to support equality deletes.
### Tests added
No new unit test is added, but the following test cases are for the change.
testEqualityDeletes, testEqualityDateDeletes, testEqualityDeletesWithRequiredEqColumn, testMixedPositionAndEqualityDeletes, testEqualityDeletesSpanningMultipleDataFiles, testMultipleEqualityDeleteSchemas, testEqualityDeleteByNull, testEqualityDeleteWithFilter, testReadEqualityDeleteRows
### Benchmark
Will add benchmark soon.
cc @rdblue @aokolnychyi, @RussellSpitzer, @karuppayya, @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r768187084
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -61,80 +62,154 @@ public void setDeleteFilter(DeleteFilter<InternalRow> deleteFilter) {
@Override
public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
- Preconditions.checkArgument(numRowsToRead > 0, "Invalid number of rows to read: %s", numRowsToRead);
- ColumnVector[] arrowColumnVectors = new ColumnVector[readers.length];
-
if (reuse == null) {
closeVectors();
}
- Pair<int[], Integer> rowIdMapping = rowIdMapping(numRowsToRead);
+ ColumnBatchLoader batchLoader = new ColumnBatchLoader(numRowsToRead);
+ rowStartPosInBatch += numRowsToRead;
+ return batchLoader.columnarBatch;
+ }
- for (int i = 0; i < readers.length; i += 1) {
- vectorHolders[i] = readers[i].read(vectorHolders[i], numRowsToRead);
- int numRowsInVector = vectorHolders[i].numValues();
- Preconditions.checkState(
- numRowsInVector == numRowsToRead,
- "Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
- numRowsToRead);
+ private class ColumnBatchLoader {
+ private int[] rowIdMapping; // the rowId mapping to skip deleted rows for all column vectors inside a batch
+ private int numRows;
+ private ColumnarBatch columnarBatch;
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
+ ColumnBatchLoader(int numRowsToRead) {
+ initRowIdMapping(numRowsToRead);
+ loadDataToColumnBatch(numRowsToRead);
+ }
+
+ ColumnarBatch loadDataToColumnBatch(int numRowsToRead) {
+ Preconditions.checkArgument(numRowsToRead > 0, "Invalid number of rows to read: %s", numRowsToRead);
+ ColumnVector[] arrowColumnVectors = readDataToColumnVectors(numRowsToRead);
+
+ columnarBatch = new ColumnarBatch(arrowColumnVectors);
+ columnarBatch.setNumRows(numRows);
+
+ if (hasEqDeletes()) {
+ applyEqDelete();
}
+ return columnarBatch;
}
- rowStartPosInBatch += numRowsToRead;
- ColumnarBatch batch = new ColumnarBatch(arrowColumnVectors);
+ ColumnVector[] readDataToColumnVectors(int numRowsToRead) {
Review comment:
This could be private too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r751708908
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -137,4 +165,30 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
return Pair.of(rowIdMapping, currentRowId);
}
}
+
+ /**
+ * Reuse the row Id mapping array to filter out equality deleted rows.
+ */
+ private void applyEqDelete(ColumnarBatch batch, int[] posDeleteRowIdMapping, int[] eqDeleteRowIdMapping) {
+ int[] rowIdMapping = posDeleteRowIdMapping == null ? eqDeleteRowIdMapping : posDeleteRowIdMapping;
Review comment:
Correct. I will move it outside of the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-993036558
> I think this is good to go, There some fields that could. be marked as private, but it's an inner class so ... it probably doesn't matter. If you are good I can merge
Thanks @RussellSpitzer for the review. It is a private internal class. Any field and method of the internal class will be public to its outer class, and private to any other classes. So the scope is the same with or without "private" modifier. It should be OK to not make them as private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766041721
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
From what I can tell, our main reason we want to divide this is that ColumnarBatchReader can't build this entire object in its constructor. So It's probably fine to keep this as an inner class of ColumnarBatchReader. I just want to keep as much state in a final constructor as possible. Maybe that's a bit weird
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766043794
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
I mean ... I think even the reader's could be passed as an arg. Then the only things that Columnar Batch Reader is responsible for is for getting the readers and start position?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r749660554
##########
File path: data/src/main/java/org/apache/iceberg/data/DeleteFilter.java
##########
@@ -192,6 +197,16 @@ protected boolean shouldKeep(T item) {
return remainingRowsFilter.filter(records);
}
+ public Predicate<T> eqDeletedRows() {
Review comment:
I'm a little confused with this naming, since I would assume "Rows" would return rows and this returns a filter basically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-972206845
> Should we have added tests her for mixed equality and positional deletes?
It is there. Check test `testMixedPositionAndEqualityDeletes`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-974498636
Resolved the comments and added the benchmark. Here is the benchmark result for 10M rows with different percentage of equality deleted rows. About 40% percent perf gain on average. Less eq delete rows, larger perf gain.
```
Benchmark (percentDeleteRow) Mode Cnt Score Error Units
IcebergSourceParquetEqDeleteBenchmark.readIceberg 0 ss 5 3.261 ± 0.210 s/op
IcebergSourceParquetEqDeleteBenchmark.readIceberg 0.000001 ss 5 3.940 ± 0.253 s/op
IcebergSourceParquetEqDeleteBenchmark.readIceberg 0.05 ss 5 4.888 ± 0.540 s/op
IcebergSourceParquetEqDeleteBenchmark.readIceberg 0.25 ss 5 7.369 ± 3.521 s/op
IcebergSourceParquetEqDeleteBenchmark.readIceberg 0.5 ss 5 9.433 ± 6.369 s/op
IcebergSourceParquetEqDeleteBenchmark.readIceberg 1 ss 5 20.206 ± 4.298 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 0 ss 5 1.721 ± 0.120 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 0.000001 ss 5 2.172 ± 0.112 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 0.05 ss 5 3.305 ± 0.301 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 0.25 ss 5 5.201 ± 3.418 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 0.5 ss 5 8.869 ± 6.236 s/op
IcebergSourceParquetEqDeleteBenchmark.readIcebergVectorized 1 ss 5 15.589 ± 7.211 s/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain edited a comment on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain edited a comment on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-972206845
> Should we have added tests her for mixed equality and positional deletes?
It is there. Check test `testMixedPositionAndEqualityDeletes`.
Thanks for the review, @RussellSpitzer. Still working on the benchmark. Will post the new commit along with the fixes for your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r768186858
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -61,80 +62,154 @@ public void setDeleteFilter(DeleteFilter<InternalRow> deleteFilter) {
@Override
public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
- Preconditions.checkArgument(numRowsToRead > 0, "Invalid number of rows to read: %s", numRowsToRead);
- ColumnVector[] arrowColumnVectors = new ColumnVector[readers.length];
-
if (reuse == null) {
closeVectors();
}
- Pair<int[], Integer> rowIdMapping = rowIdMapping(numRowsToRead);
+ ColumnBatchLoader batchLoader = new ColumnBatchLoader(numRowsToRead);
+ rowStartPosInBatch += numRowsToRead;
+ return batchLoader.columnarBatch;
+ }
- for (int i = 0; i < readers.length; i += 1) {
- vectorHolders[i] = readers[i].read(vectorHolders[i], numRowsToRead);
- int numRowsInVector = vectorHolders[i].numValues();
- Preconditions.checkState(
- numRowsInVector == numRowsToRead,
- "Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
- numRowsToRead);
+ private class ColumnBatchLoader {
+ private int[] rowIdMapping; // the rowId mapping to skip deleted rows for all column vectors inside a batch
+ private int numRows;
+ private ColumnarBatch columnarBatch;
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
+ ColumnBatchLoader(int numRowsToRead) {
+ initRowIdMapping(numRowsToRead);
+ loadDataToColumnBatch(numRowsToRead);
+ }
+
+ ColumnarBatch loadDataToColumnBatch(int numRowsToRead) {
Review comment:
private right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer merged pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer merged pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r764313831
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -34,6 +35,7 @@
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.jetbrains.annotations.Nullable;
Review comment:
Probably this should be the javax.annotation.Nullable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-993150790
Thanks @flyrain ! You are really taking the vectorization support to a new level here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766000770
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
I experimented that way. It turned out almost nothing left in the class `ColumnBatchReader`. So I asked myself why I should have another class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766043794
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
I mean ... I think even the reader's could be passed as an arg. Then the only things that Columnar Batch Reader is responsible for is for getting the readers and num rows to read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-993838392
Thanks a lot for the review, @RussellSpitzer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#issuecomment-970746207
Should we have added tests her for mixed equality and positional deletes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
flyrain commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766214001
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
Changed it to an inner class and moved vector read logic into the inner class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r765984452
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
I think I would pass the vectors directly into the BatchWithFilte so everything like this check can be done inside that class as well. Then this class just creates the the BatchWith Filter with the proper vectors and then call .batch which would return a final columnar Batch from inside the class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r765392034
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnBatchWithRowMapping.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.data.vectorized;
+
+import java.util.Iterator;
+import org.apache.iceberg.data.DeleteFilter;
+import org.apache.iceberg.deletes.PositionDeleteIndex;
+import org.apache.iceberg.util.Pair;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.vectorized.ColumnVector;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+class ColumnBatchWithRowMapping {
+ private final DeleteFilter<InternalRow> deletes;
+ private int[] rowIdMapping; // the rowId mapping to skip deleted rows for all column vectors inside a batch
+ private int numRows;
+ private ColumnarBatch batch;
+
+ ColumnBatchWithRowMapping(DeleteFilter<InternalRow> deletes, int numRowsToRead, long rowStartPosInBatch) {
+ this.deletes = deletes;
+ initRowIdMapping(numRowsToRead, rowStartPosInBatch);
+ }
+
+ ColumnarBatch createColumnBatch(ColumnVector[] columns) {
+ batch = new ColumnarBatch(columns);
+ batch.setNumRows(numRows);
+
+ if (hasEqDeletes()) {
+ applyEqDelete();
+ }
+ return batch;
+ }
+
+ boolean hasDeletes() {
+ return rowIdMapping != null;
+ }
+
+ private boolean hasEqDeletes() {
+ return deletes != null && deletes.hasEqDeletes();
+ }
+
+ int[] rowIdMapping() {
+ return rowIdMapping;
+ }
+
+ int numRows() {
+ return numRows;
+ }
+
+ private void initRowIdMapping(int numRowsToRead, long rowStartPosInBatch) {
+ Pair<int[], Integer> posDeleteRowIdMapping = posDelRowIdMapping(numRowsToRead, rowStartPosInBatch);
+ if (posDeleteRowIdMapping != null) {
+ rowIdMapping = posDeleteRowIdMapping.first();
+ numRows = posDeleteRowIdMapping.second();
+ } else {
+ numRows = numRowsToRead;
+ rowIdMapping = initEqDeleteRowIdMapping(numRowsToRead);
+ }
+ }
+
+ private Pair<int[], Integer> posDelRowIdMapping(int numRowsToRead, long rowStartPosInBatch) {
+ if (deletes != null && deletes.hasPosDeletes()) {
+ return buildPosDelRowIdMapping(deletes.deletedRowPositions(), numRowsToRead, rowStartPosInBatch);
+ } else {
+ return null;
+ }
+ }
+
+ /**
+ * Build a row id mapping inside a batch, which skips delete rows. For example, if the 1st and 3rd rows are deleted in
Review comment:
"skips delete" -> "skips deleted"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r766041721
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -78,63 +76,13 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
"Number of rows in the vector %s didn't match expected %s ", numRowsInVector,
numRowsToRead);
- if (rowIdMapping == null) {
- arrowColumnVectors[i] = IcebergArrowColumnVector.forHolder(vectorHolders[i], numRowsInVector);
- } else {
- int[] rowIdMap = rowIdMapping.first();
- Integer numRows = rowIdMapping.second();
- arrowColumnVectors[i] = ColumnVectorWithFilter.forHolder(vectorHolders[i], rowIdMap, numRows);
- }
+ arrowColumnVectors[i] = batch.hasDeletes() ?
Review comment:
From what I can tell, our main reason we want to divide this is that ColumnarBatchReader can't build this entire object in its constructor. So It's probably fine to keep this as an inner class of ColumnarBatchReader. I just want to keep as much state in a final constructor as possible. Maybe that's a bit weird so if you have other feelings let me know. But basically I was hoping all of the more complicated state gets created inside an immutable class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r750719965
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -137,4 +165,30 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
return Pair.of(rowIdMapping, currentRowId);
}
}
+
+ /**
+ * Reuse the row Id mapping array to filter out equality deleted rows.
+ */
+ private void applyEqDelete(ColumnarBatch batch, int[] posDeleteRowIdMapping, int[] eqDeleteRowIdMapping) {
+ int[] rowIdMapping = posDeleteRowIdMapping == null ? eqDeleteRowIdMapping : posDeleteRowIdMapping;
Review comment:
It seems like this will only respect one of the two mappings? Or am I wrong there? In general I would try to avoid functions which only use some arguments if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3557: Spark: Support vectorized reads with equality deletes
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #3557:
URL: https://github.com/apache/iceberg/pull/3557#discussion_r750721112
##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java
##########
@@ -96,9 +94,39 @@ public final ColumnarBatch read(ColumnarBatch reuse, int numRowsToRead) {
Integer numRows = rowIdMapping.second();
batch.setNumRows(numRows);
}
+
+ if (deletes != null && deletes.hasEqDeletes()) {
+ applyEqDelete(batch, rowIdMapping == null ? null : rowIdMapping.first(), eqDeleteRowIdMapping);
Review comment:
I would probably try to extract out the logic for choosing which row mapping to use from "applyEqDelete" and do it here instead so it's simpler to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org