[I] Native Iceberg write fails with "No column with source column id" after an identity partition field is dropped [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #5693:
URL: https://github.com/apache/datafusion-comet/issues/5693

   ### Describe the bug
   
   After an identity partition field is dropped and a column is re-added, the native writer cannot resolve the partition source column and fails the write:
   
   ```
   org.apache.comet.CometNativeException: Unexpected => No column with source column id 2 in schema
     Schema { struct: StructType { fields: [NestedField { id: 1, name: "id", required: true,
     field_type: Primitive(Long), ... } ...
       at org.apache.comet.Native.executePlan(Native Method)
       at org.apache.spark.sql.comet.CometIcebergWriteExec.drainAvroPayload(CometIcebergWriteExec.scala:276)
       at org.apache.spark.sql.comet.CometIcebergWriteExec.$anonfun$doExecute$9(CometIcebergWriteExec.scala:181)
   ```
   
   The partition spec still references source column id 2, which no longer exists in the current schema, and the native manifest encoding treats that as fatal. iceberg-java tolerates it.
   
   ### Steps to reproduce
   
   Spark 4.1.3, Iceberg 1.11.0, `spark.comet.iceberg.write.enabled=true` plus the Iceberg Spark SQL test setup from `dev/diffs/iceberg/1.11.0.diff`:
   
   ```
   ./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 -DflinkVersions= -DkafkaVersions= \
     :iceberg-spark:iceberg-spark-extensions-4.1_2.13:test \
     --tests '*TestAlterTablePartitionFields*' -Pquick=true -x javadoc
   ```
   
   Failing case: `testReaddColumnAfterIdentityPartitionDrop`.
   
   ### Expected behavior
   
   The write succeeds against a table whose partition spec references a source column that is no longer in the current schema, matching iceberg-java, or the operator declines the write at planning time so it falls back to the JVM writer instead of failing the task.
   
   ### Additional context
   
   Found by turning the two Iceberg write flags on by default in #5677.
   
   Same test class and same `encode_data_files_as_manifest` phase as #5691, which panics on a related partition-spec-evolution case. One fix to how the writer resolves the spec against the current schema may close both.
   
   Part of #5649.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Native Iceberg write fails with "No column with source column id" after an identity partition field is dropped [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #5693: Native Iceberg write fails with "No column with source column id" after an identity partition field is dropped
URL: https://github.com/apache/datafusion-comet/issues/5693


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org