You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "jinchengchenghh (via GitHub)" <gi...@apache.org> on 2025/03/10 10:10:51 UTC
[I] Iceberg delete from throws exception [incubator-gluten]
jinchengchenghh opened a new issue, #8948:
URL: https://github.com/apache/incubator-gluten/issues/8948
### Backend
VL (Velox)
### Bug description
Reproduce,
IcebergSuite, remove the GLUTEN_ENABLED false in delete sql block
GlutenTestMergeOnReadDelete#testAggregatePushDownInMergeOnReadDelete,
If fallback by config `spark.gluten.sql.columnar.batchscan=false`, the tests can pass.
Maybe the IcebergScanTransformer has some hidden bugs.
```
Caused by: java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$6(ShuffleExchangeExec.scala:317)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$6$adapted(ShuffleExchangeExec.scala:317)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$14(ShuffleExchangeExec.scala:386)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:169)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
```
### Spark version
Spark-3.4.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Iceberg delete from throws exception [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8948:
URL: https://github.com/apache/incubator-gluten/issues/8948#issuecomment-2724831420
There is also `DeletedVectorReader`, `PositionVectorReader` which is different with common data columns reader, maybe we also need to specialize it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Iceberg delete from throws exception [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8948:
URL: https://github.com/apache/incubator-gluten/issues/8948#issuecomment-2743962645
fallbacked now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Iceberg delete from throws exception [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8948:
URL: https://github.com/apache/incubator-gluten/issues/8948#issuecomment-2724825095
Delete scan reads the metadata column `_file`, the information is not in velox.
https://github.com/apache/iceberg/blob/main/spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/BatchDataReader.java#L98
In iceberg, the data file reader create a constant column for the metadata columns,
```
public static Map<Integer, ?> constantsMap(
ContentScanTask<?> task,
Types.StructType partitionType,
BiFunction<Type, Object, Object> convertConstant) {
PartitionSpec spec = task.spec();
StructLike partitionData = task.file().partition();
// use java.util.HashMap because partition data may contain null values
Map<Integer, Object> idToConstant = Maps.newHashMap();
// add _file
idToConstant.put(
MetadataColumns.FILE_PATH.fieldId(),
convertConstant.apply(Types.StringType.get(), task.file().path()));
// add _spec_id
idToConstant.put(
MetadataColumns.SPEC_ID.fieldId(),
convertConstant.apply(Types.IntegerType.get(), task.file().specId()));
// add _partition
if (partitionType != null) {
if (!partitionType.fields().isEmpty()) {
StructLike coercedPartition = coercePartition(partitionType, spec, partitionData);
idToConstant.put(
MetadataColumns.PARTITION_COLUMN_ID,
convertConstant.apply(partitionType, coercedPartition));
} else {
// use null as some query engines may not be able to handle empty structs
idToConstant.put(MetadataColumns.PARTITION_COLUMN_ID, null);
}
}
```
And the PARTITION_COLUMN_ID is also constant column.
So we need to extract the metadata column, only query the data columns from velox and add the _delete or _file flag as extra constant Vector to data RowVector.
_delete is used in equality delete file reader test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Iceberg delete from throws exception [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh closed issue #8948: [VL] Iceberg delete from throws exception
URL: https://github.com/apache/incubator-gluten/issues/8948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Iceberg delete from throws exception [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh closed issue #8948: [VL] Iceberg delete from throws exception
URL: https://github.com/apache/incubator-gluten/issues/8948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org