You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2026/01/28 05:54:06 UTC

[I] deltalake partitioned write still hass a C2R converter [incubator-gluten]

FelixYBW opened a new issue, #11503:
URL: https://github.com/apache/incubator-gluten/issues/11503

   ### Description
   
   there still is a C2R in the parquet write stage when we create partitioned table like
   
   ```python
   df.write.mode("overwrite").format("delta").option("compression", "zstd").partitionBy(tbl_filenum[tbl]['part']).option("optimizeWrite", "True").save(f"s3a://presto-workload/{databasename}/{tbl}")
   ```
   
   <img width="663" height="362" alt="Image" src="https://github.com/user-attachments/assets/82a6080f-61aa-4a48-a897-2c6f9fb73a20" />
   
   saw in call stack:
   ```
   app//org.apache.gluten.vectorized.NativeColumnarToRowJniWrapper.nativeColumnarToRowConvert(Native Method)
   app//org.apache.gluten.execution.VeloxColumnarToRowExec$Converter$$anon$2.next(VeloxColumnarToRowExec.scala:193)
   app//org.apache.gluten.execution.VeloxColumnarToRowExec$Converter$$anon$2.next(VeloxColumnarToRowExec.scala:169)
   app//scala.collection.Iterator.foreach(Iterator.scala:943)
   app//scala.collection.Iterator.foreach$(Iterator.scala:943)
   app//org.apache.gluten.execution.VeloxColumnarToRowExec$Converter$$anon$2.foreach(VeloxColumnarToRowExec.scala:169)
   app//org.apache.spark.sql.delta.stats.GlutenDeltaJobStatisticsTracker$GlutenDeltaTaskStatisticsTracker.newRow(GlutenDeltaWriteJobStatsTracker.scala:73)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$ColumnarDynamicPartitionDataSingleWriter.$anonfun$write$11(GlutenDeltaFileFormatWriter.scala:596)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$ColumnarDynamicPartitionDataSingleWriter.$anonfun$write$11$adapted(GlutenDeltaFileFormatWriter.scala:595)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$ColumnarDynamicPartitionDataSingleWriter$$Lambda$2037/0x00000008018bc318.apply(Unknown Source)
   app//scala.collection.immutable.List.foreach(List.scala:431)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$ColumnarDynamicPartitionDataSingleWriter.write(GlutenDeltaFileFormatWriter.scala:595)
   app//org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
   app//org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$.$anonfun$executeTask$1(GlutenDeltaFileFormatWriter.scala:478)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$$$Lambda$1916/0x00000008018877f0.apply(Unknown Source)
   app//org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$.executeTask(GlutenDeltaFileFormatWriter.scala:485)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$.$anonfun$executeWrite$4(GlutenDeltaFileFormatWriter.scala:313)
   app//org.apache.spark.sql.delta.files.GlutenDeltaFileFormatWriter$$$Lambda$1553/0x00000008016bd7f0.apply(Unknown Source)
   app//org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
   app//org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
   app//org.apache.spark.scheduler.Task.run(Task.scala:141)
   app//org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
   app//org.apache.spark.executor.Executor$TaskRunner$$Lambda$936/0x00000008012177d8.apply(Unknown Source)
   app//org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
   app//org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
   app//org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
   app//org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
   java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
   ```
   
   ### Gluten version
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] deltalake partitioned write still hass a C2R converter [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #11503:
URL: https://github.com/apache/incubator-gluten/issues/11503#issuecomment-3810198967

   `VeloxColumnarToCarrierRow` is a fake C2R operator that doesn't have high cost as regular C2R.
   
   However, there is a C2R during writing the Delta statistics files. I am working on removing that one https://github.com/apache/incubator-gluten/pull/11419.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] deltalake partitioned write still hass a C2R converter [gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #11503: [VL] deltalake partitioned write still hass a C2R converter
URL: https://github.com/apache/gluten/issues/11503


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] deltalake partitioned write still hass a C2R converter [gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #11503:
URL: https://github.com/apache/gluten/issues/11503#issuecomment-4114133920

   close as finished


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org