You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huaxingao (via GitHub)" <gi...@apache.org> on 2024/11/14 16:57:44 UTC

[PR] Spark: Remove extra columns for ColumnBatch [iceberg]

huaxingao opened a new pull request, #11551:
URL: https://github.com/apache/iceberg/pull/11551

   In Equality Delete, we build `ColumnarBatchReader` for the equality delete filter columns to read their values and determine which rows are deleted. If these filter columns are not among the requested columns, they are considered extra and should be removed before returning the `ColumnBatch` to Spark.
   
   Suppose the table schema includes C1, C2, C3, C4, C5. If the query is: `SELECT C5 FROM table`, and the equality delete filter is on C3 and C4,
   
   We read the values of C3 and C4 to identify which rows are deleted. However, we do not want to include these values in the `ColumnBatch` that we return to Spark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thank you for the response !
   
   Considering applyEquality delete anyways does another projection on top of the schema returned from DeleteFilte.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)
   
   https://github.com/apache/iceberg/blob/bf8d25fe1578ef199d64fb609c0299728ec58910/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L196
   
   can we not add one another param in the fileProjection like we did here to include additional fiels based on the boolean flag ? : 
   https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272 
   
   so that we get what columns we actually need in the first place ? to avoid removing extra columns post filter evaluation 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1843189307


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thanks for your comment!
   
    The extra columns are appended to the requested schema in DeleteFilter.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306). The values of these extra columns are read in ColumnarBatchReader and used to identify which rows are deleted in [applyEqDelete.](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java#L244) I remove the extra columns right after calling applyEqDelete.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2512581027

   @flyrain I think this over. The `missingIds` could be from [`ROW_POSITION.fieldId()`](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L273C39-L273C61) or from [`eqDelete.equalityFieldIds()`](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L277C26-L277C53). In batch case, [`needRowPosCol`](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272C9-L272C22) is false, so we don't have `ROW_POSITION`. Is it possible to have `IS_DELETED.fieldId()` in `eqDelete.equalityFieldIds()`, meaning, is it possible to do an equality delete based on `IS_DELETED` column? If not, then `missingIds` should not contain `IS_DELETED.fieldId()`.
   
   If we explicitly select `_delete`, then the `_delete` is in the `expectedSchema`. For example, if the table has c1 and c2, and equality delete is based on c1, and we run SELECT c2 , _delete FROM table, the `expectedSchema` is c2, _delete, and the missing column is c1. The `requiredSchema` would then be c2, _delete, c1, thus allowing us to still remove the extra column from the end of the columns.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865274246


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +247,16 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumns(

Review Comment:
   It would be helpful to keep the following comments as part of the JavaDoc for the method to provide clarity and context:
   ```
       /**
        * Removes extra columns added for processing equality delete filters that are not part of the
        * final query output.
        * <p>
        * During query execution, additional columns may be included in the schema to evaluate equality
        * delete filters. For example, if the table schema contains columns C1, C2, C3, C4, and C5, and
        * the query is 'SELECT C5 FROM table' while equality delete filters are applied on C3 and C4,
        * the processing schema includes C5, C3, and C4. These extra columns (C3 and C4) are needed
        * to identify rows to delete but are not included in the final result.
        * <p>
        * This method removes these extra columns from the end of {@code arrowColumnVectors}, ensuring
        * only the expected columns remain.
        *
        * @param arrowColumnVectors the array of column vectors representing query result data
        * @param columnarBatch the original {@code ColumnarBatch} containing query results
        * @return a new {@code ColumnarBatch} with extra columns removed, or the original batch if no
        *         extra columns were found
        */
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2512615952

   > In batch case, [needRowPosCol](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272C9-L272C22) is false
   
   Is there a need to fix the non-batch use cases?
   
   > Is it possible to have IS_DELETED.fieldId() in eqDelete.equalityFieldIds(), meaning, is it possible to do an equality delete based on IS_DELETED column?
   
   Should be possible.
   
   For the batch use cases, I was thinking about a more generic solution, which we can check the number of columns in `ColumnarBatch` is larger than the number of expected columns. If yes, remove the rest, otherwise do nothing. Basically do the same thing in the current PR, but not limited to eq deletes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain merged PR #11551:
URL: https://github.com/apache/iceberg/pull/11551


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thank you for the response !
   
   [doubt] Considering applyEquality delete anyways does another projection on top of the schema returned from DeleteFilte.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)
   
   https://github.com/apache/iceberg/blob/bf8d25fe1578ef199d64fb609c0299728ec58910/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L196
   
   can we not add one another param in the fileProjection like we did here to include additional fiels based on the boolean flag ? : 
   https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272 
   
   so that we get what columns we actually need in the first place ? to avoid removing extra columns post filter evaluation 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1842720884


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   [doubt] Is it possible to fix this at the place where these extra columns are appended in the end of the requested schema, this would probably help us in avoiding extra memory in the first place and expensive copy of Columbatch ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2479588343

   cc @flyrain @szehon-ho @viirya 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865218949


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -622,6 +623,50 @@ public void testPosDeletesOnParquetFileWithMultipleRowGroups() throws IOExceptio
     assertThat(rowSet(tblName, tbl, "*")).hasSize(193);
   }
 
+  @TestTemplate
+  public void testEqualityDeleteWithDifferentScanAndDeleteColumns() throws IOException {
+    assumeThat(format).isEqualTo(FileFormat.PARQUET);
+    initDateTable();
+
+    Schema deleteRowSchema = dateTable.schema().select("dt");
+    Record dataDelete = GenericRecord.create(deleteRowSchema);
+    List<Record> dataDeletes =
+        Lists.newArrayList(
+            dataDelete.copy("dt", LocalDate.parse("2021-09-01")),
+            dataDelete.copy("dt", LocalDate.parse("2021-09-02")),
+            dataDelete.copy("dt", LocalDate.parse("2021-09-03")));
+
+    DeleteFile eqDeletes =
+        FileHelpers.writeDeleteFile(
+            dateTable,
+            Files.localOutput(File.createTempFile("junit", null, temp.toFile())),
+            TestHelpers.Row.of(0),
+            dataDeletes.subList(0, 3),
+            deleteRowSchema);
+
+    dateTable.newRowDelta().addDeletes(eqDeletes).commit();
+
+    CloseableIterable<CombinedScanTask> tasks =
+        TableScanUtil.planTasks(
+            dateTable.newScan().planFiles(),
+            TableProperties.METADATA_SPLIT_SIZE_DEFAULT,
+            TableProperties.SPLIT_LOOKBACK_DEFAULT,
+            TableProperties.SPLIT_OPEN_FILE_COST_DEFAULT);
+
+    for (CombinedScanTask task : tasks) {
+      try (BatchDataReader reader =
+          new BatchDataReader(
+              // expected column is id, while the equality filter column is dt
+              dateTable, task, dateTable.schema(), dateTable.schema().select("id"), false, 7)) {
+        while (reader.next()) {
+          ColumnarBatch columnarBatch = reader.get();
+          int numOfCols = columnarBatch.numCols();
+          assertThat(numOfCols).as("Number of columns").isEqualTo(1);

Review Comment:
   Minor: also check the column type to make sure `dt` is removed like following?
   ```
             // only the expected column(id) is kept
             assertThat(columnarBatch.numCols()).as("Number of columns").isEqualTo(1);
             assertThat(columnarBatch.column(0).dataType()).as("Column type").isEqualTo(IntegerType);
   ```



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +269,20 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumns(
+        ColumnVector[] arrowColumnVectors, ColumnarBatch columnarBatch) {
+      int numOfExtraColumns = numOfExtraColumns(deletes);
+      if (numOfExtraColumns > 0) {
+        int newLength = arrowColumnVectors.length - numOfExtraColumns;
+        // In DeleteFilter.fileProjection, the columns for missingIds (the columns required
+        // for equality delete or ROW_POSITION) are appended to the end of the expectedSchema.
+        // Therefore, these extra columns can be removed from the end of arrowColumnVectors.
+        ColumnVector[] newColumns = Arrays.copyOf(arrowColumnVectors, newLength);
+        return new ColumnarBatch(newColumns, columnarBatch.numRows());
+      } else {
+        return columnarBatch;
+      }
+    }

Review Comment:
   How about refactor it like this,  so that method `numOfExtraColumns` is not needed? Can we also move all related comments to this method's Java doc?
   ```
         int expectedColumnSize = deletes.expectedSchema().columns().size();
         if (arrowColumnVectors.length > expectedColumnSize) {
           ColumnVector[] newColumns = Arrays.copyOf(arrowColumnVectors, expectedColumnSize);
           return new ColumnarBatch(newColumns, columnarBatch.numRows());
         } else {
           return columnarBatch;
         }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2512236843

   @flyrain I will have a follow-up PR. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2513346295

   Thanks @flyrain for reviewing and merging the PR! Also thanks @singhpk234 for reviewing!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thank you for the response !
   
   Considering applyEquality delete anyways does another projection on top of the schema returned from DeleteFilte.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)
   
   https://github.com/apache/iceberg/blob/bf8d25fe1578ef199d64fb609c0299728ec58910/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L196
   
   can we not add one another param in the fileProjection like we did here to include additional fiels based on the boolean flag : 
   https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272 
   



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thank you for the response !
   
   Considering applyEquality delete anyways does another projection on top of the schema returned from DeleteFilte.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)
   
   https://github.com/apache/iceberg/blob/bf8d25fe1578ef199d64fb609c0299728ec58910/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L196
   
   can we not add one another param in the fileProjection like we did here to include additional fiels based on the boolean flag ? : 
   https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L272 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1863894343


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +259,16 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumnsFromColumnarBatch(

Review Comment:
   Minor suggestion: Simplify to `removeExtraColumns`?



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.
+  private int numOfExtraColumns = 0;
 
-  public ColumnarBatchReader(List<VectorizedReader<?>> readers) {
+  public ColumnarBatchReader(List<VectorizedReader<?>> readers, int numExtraCol) {

Review Comment:
   Can we use the DeleteFilter(field `deletes`) within the class, so that no extra parameter is required? We could move the method `numOfExtraColumns` to this class in that case.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {

Review Comment:
   minor: `DeleteFilter` -> `dDeleteFilter<InternalRow>`



##########
data/src/main/java/org/apache/iceberg/data/DeleteFilter.java:
##########
@@ -73,6 +74,7 @@ protected DeleteFilter(
       boolean needRowPosCol) {
     this.filePath = filePath;
     this.counter = counter;
+    this.requestedSchema = requestedSchema;

Review Comment:
   Can we be consistent with the name? `requestedSchema` or `expectedSchema`? I guess `expectedSchema` is more commonly used.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {
+    if (deleteFilter != null) {
+      if (deleteFilter.hasEqDeletes()) {
+        // For Equality Delete, the requiredColumns and expectedColumns may not be the
+        // same. For example, supposed table schema is C1, C2, C3, C4, C5, The query is:
+        // SELECT C5 FROM table, and the equality delete Filter is on C3, C4, then
+        // the requestedSchema is C5, and the required schema is C5, C3 and C4. The
+        // vectorized reader reads also need to read C3 and C4 columns to figure out
+        // which rows are deleted. However, after figuring out the deleted rows, the
+        // extra columns values are not needed to returned to Spark.
+        // Getting the numOfExtraColumns so we can remove these extra columns
+        // from ColumnBatch later.

Review Comment:
   Can we move the comment to method's Java doc?



##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -622,6 +624,41 @@ public void testPosDeletesOnParquetFileWithMultipleRowGroups() throws IOExceptio
     assertThat(rowSet(tblName, tbl, "*")).hasSize(193);
   }
 
+  @TestTemplate
+  public void testEqualityDeleteWithDifferentScanAndDeleteColumns() throws IOException {

Review Comment:
   Is it possible to check the intermediate results? for example, checking the `ColumnarBatch` returned to Spark. We may avoid using `comet` as a dependency for the test.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {
+    if (deleteFilter != null) {
+      if (deleteFilter.hasEqDeletes()) {
+        // For Equality Delete, the requiredColumns and expectedColumns may not be the
+        // same. For example, supposed table schema is C1, C2, C3, C4, C5, The query is:
+        // SELECT C5 FROM table, and the equality delete Filter is on C3, C4, then
+        // the requestedSchema is C5, and the required schema is C5, C3 and C4. The
+        // vectorized reader reads also need to read C3 and C4 columns to figure out
+        // which rows are deleted. However, after figuring out the deleted rows, the
+        // extra columns values are not needed to returned to Spark.
+        // Getting the numOfExtraColumns so we can remove these extra columns
+        // from ColumnBatch later.
+        List<Types.NestedField> requiredColumns = deleteFilter.requiredSchema().columns();
+        List<Types.NestedField> expectedColumns = deleteFilter.requestedSchema().columns();
+        return requiredColumns.size() - expectedColumns.size();

Review Comment:
   Are we sure the extra columns are consistently appended to the end of the array?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865324477


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +247,16 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumns(

Review Comment:
   Sorry, missed this part. Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1864760757


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {

Review Comment:
   Fixed. Thanks



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {
+    if (deleteFilter != null) {
+      if (deleteFilter.hasEqDeletes()) {
+        // For Equality Delete, the requiredColumns and expectedColumns may not be the
+        // same. For example, supposed table schema is C1, C2, C3, C4, C5, The query is:
+        // SELECT C5 FROM table, and the equality delete Filter is on C3, C4, then
+        // the requestedSchema is C5, and the required schema is C5, C3 and C4. The
+        // vectorized reader reads also need to read C3 and C4 columns to figure out
+        // which rows are deleted. However, after figuring out the deleted rows, the
+        // extra columns values are not needed to returned to Spark.
+        // Getting the numOfExtraColumns so we can remove these extra columns
+        // from ColumnBatch later.

Review Comment:
   Moved. Thanks



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:
##########
@@ -125,4 +126,25 @@ protected VectorizedReader<?> vectorizedReader(List<VectorizedReader<?>> reorder
       return reader;
     }
   }
+
+  private static int numOfExtraColumns(DeleteFilter deleteFilter) {
+    if (deleteFilter != null) {
+      if (deleteFilter.hasEqDeletes()) {
+        // For Equality Delete, the requiredColumns and expectedColumns may not be the
+        // same. For example, supposed table schema is C1, C2, C3, C4, C5, The query is:
+        // SELECT C5 FROM table, and the equality delete Filter is on C3, C4, then
+        // the requestedSchema is C5, and the required schema is C5, C3 and C4. The
+        // vectorized reader reads also need to read C3 and C4 columns to figure out
+        // which rows are deleted. However, after figuring out the deleted rows, the
+        // extra columns values are not needed to returned to Spark.
+        // Getting the numOfExtraColumns so we can remove these extra columns
+        // from ColumnBatch later.
+        List<Types.NestedField> requiredColumns = deleteFilter.requiredSchema().columns();
+        List<Types.NestedField> expectedColumns = deleteFilter.requestedSchema().columns();
+        return requiredColumns.size() - expectedColumns.size();

Review Comment:
   Yes, because the extra columns are appended to the end of `requestedSchema` in [DeleteFilter.fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1864760863


##########
data/src/main/java/org/apache/iceberg/data/DeleteFilter.java:
##########
@@ -73,6 +74,7 @@ protected DeleteFilter(
       boolean needRowPosCol) {
     this.filePath = filePath;
     this.counter = counter;
+    this.requestedSchema = requestedSchema;

Review Comment:
   Changed to `expectedSchema`



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.
+  private int numOfExtraColumns = 0;
 
-  public ColumnarBatchReader(List<VectorizedReader<?>> readers) {
+  public ColumnarBatchReader(List<VectorizedReader<?>> readers, int numExtraCol) {

Review Comment:
   Changed. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1864760687


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -622,6 +624,41 @@ public void testPosDeletesOnParquetFileWithMultipleRowGroups() throws IOExceptio
     assertThat(rowSet(tblName, tbl, "*")).hasSize(193);
   }
 
+  @TestTemplate
+  public void testEqualityDeleteWithDifferentScanAndDeleteColumns() throws IOException {

Review Comment:
   I have changed the test to check the number of columns in `ColumnarBatch`



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +259,16 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumnsFromColumnarBatch(

Review Comment:
   Done. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2512632326

   @flyrain Thanks for the quick reply. I will have a follow-up PR for this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1842606127


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -622,6 +624,41 @@ public void testPosDeletesOnParquetFileWithMultipleRowGroups() throws IOExceptio
     assertThat(rowSet(tblName, tbl, "*")).hasSize(193);
   }
 
+  @TestTemplate
+  public void testEqualityDeleteWithDifferentScanAndDeleteColumns() throws IOException {

Review Comment:
   This test is expected to pass even without the fix provided by this PR. Currently, the extra columns returned to Spark do not cause any problems. However, with [Comet](https://github.com/apache/iceberg/pull/9841) native execution, since Comet allocates arrays in a pre-allocated list and relies on the requested schema to determine the number of columns in the batch, this test would fail without the fix proposed in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1842720884


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   [doubt] Is it possible to fix this at the place where these extra columns are appended in the end of the requested schema, this would probably help us in avoiding extra memory in the first place and expensive copy of command ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847131830


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -45,11 +45,23 @@ public class ColumnarBatchReader extends BaseBatchReader<ColumnarBatch> {
   private final boolean hasIsDeletedColumn;
   private DeleteFilter<InternalRow> deletes = null;
   private long rowStartPosInBatch = 0;
+  // In the case of Equality Delete, we have also built ColumnarBatchReader for the equality delete
+  // filter columns to read the value to find out which rows are deleted. If these deleted filter
+  // columns are not in the requested schema, then these are the extra columns that we want to
+  // remove before return the ColumnBatch to Spark.
+  // Supposed table schema is C1, C2, C3, C4, C5, The query is:
+  // SELECT C5 FROM table, and the equality delete Filter is on C3, C4,
+  // We read the values of C3, C4 to figure out which rows are deleted, but we don't want to include
+  // these values in the ColumnBatch that we return to Spark. In this example, the numOfExtraColumns
+  // is 2. Since when creating the DeleteFilter, we append these extra columns in the end of the
+  // requested schema, we can just remove them from the end of the ColumnVector.

Review Comment:
   Thanks for taking a look!
   We traverse the schema and build a `VectorizedReader` for each of the column in [VectorizedReaderBuilder](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java#L71), this is done before DeleteFilte.[fileProjection](https://github.com/apache/iceberg/blob/06dc721498d6ad95c86f0f884b8ad30f807ef321/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L306)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #11551:
URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865249456


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -622,6 +623,50 @@ public void testPosDeletesOnParquetFileWithMultipleRowGroups() throws IOExceptio
     assertThat(rowSet(tblName, tbl, "*")).hasSize(193);
   }
 
+  @TestTemplate
+  public void testEqualityDeleteWithDifferentScanAndDeleteColumns() throws IOException {
+    assumeThat(format).isEqualTo(FileFormat.PARQUET);
+    initDateTable();
+
+    Schema deleteRowSchema = dateTable.schema().select("dt");
+    Record dataDelete = GenericRecord.create(deleteRowSchema);
+    List<Record> dataDeletes =
+        Lists.newArrayList(
+            dataDelete.copy("dt", LocalDate.parse("2021-09-01")),
+            dataDelete.copy("dt", LocalDate.parse("2021-09-02")),
+            dataDelete.copy("dt", LocalDate.parse("2021-09-03")));
+
+    DeleteFile eqDeletes =
+        FileHelpers.writeDeleteFile(
+            dateTable,
+            Files.localOutput(File.createTempFile("junit", null, temp.toFile())),
+            TestHelpers.Row.of(0),
+            dataDeletes.subList(0, 3),
+            deleteRowSchema);
+
+    dateTable.newRowDelta().addDeletes(eqDeletes).commit();
+
+    CloseableIterable<CombinedScanTask> tasks =
+        TableScanUtil.planTasks(
+            dateTable.newScan().planFiles(),
+            TableProperties.METADATA_SPLIT_SIZE_DEFAULT,
+            TableProperties.SPLIT_LOOKBACK_DEFAULT,
+            TableProperties.SPLIT_OPEN_FILE_COST_DEFAULT);
+
+    for (CombinedScanTask task : tasks) {
+      try (BatchDataReader reader =
+          new BatchDataReader(
+              // expected column is id, while the equality filter column is dt
+              dateTable, task, dateTable.schema(), dateTable.schema().select("id"), false, 7)) {
+        while (reader.next()) {
+          ColumnarBatch columnarBatch = reader.get();
+          int numOfCols = columnarBatch.numCols();
+          assertThat(numOfCols).as("Number of columns").isEqualTo(1);

Review Comment:
   Added. Thanks!



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java:
##########
@@ -245,5 +269,20 @@ void applyEqDelete(ColumnarBatch columnarBatch) {
 
       columnarBatch.setNumRows(currentRowId);
     }
+
+    ColumnarBatch removeExtraColumns(
+        ColumnVector[] arrowColumnVectors, ColumnarBatch columnarBatch) {
+      int numOfExtraColumns = numOfExtraColumns(deletes);
+      if (numOfExtraColumns > 0) {
+        int newLength = arrowColumnVectors.length - numOfExtraColumns;
+        // In DeleteFilter.fileProjection, the columns for missingIds (the columns required
+        // for equality delete or ROW_POSITION) are appended to the end of the expectedSchema.
+        // Therefore, these extra columns can be removed from the end of arrowColumnVectors.
+        ColumnVector[] newColumns = Arrays.copyOf(arrowColumnVectors, newLength);
+        return new ColumnarBatch(newColumns, columnarBatch.numRows());
+      } else {
+        return columnarBatch;
+      }
+    }

Review Comment:
   Good suggestion! Changed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org