You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/12/19 17:13:17 UTC
[I] Comet fails when local writer enabled [datafusion-comet]
comphead opened a new issue, #2944:
URL: https://github.com/apache/datafusion-comet/issues/2944
### Describe the bug
```
spark.range(1000).write.mode("overwrite").parquet("/tmp/writertest/input")
```
Caused by: org.apache.spark.SparkException: Comet execution only takes Arrow Arrays, but got class org.apache.spark.sql.execution.vectorized.OnHeapColumnVector
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native writer crashes for non Arrow input data [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2944: Comet native writer crashes for non Arrow input data
URL: https://github.com/apache/datafusion-comet/issues/2944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet fails when local writer enabled [datafusion-comet]
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on issue #2944:
URL: https://github.com/apache/datafusion-comet/issues/2944#issuecomment-3739041699
Looks like this is coming from https://github.com/apache/datafusion-comet/blob/3a6452b3aee2620146c029e5f156031c7c7937dc/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala#L146
As comment suggest, let me try converting Spark batches to Arrow format using CometArrowConverters.columnarBatchToArrowBatchIter()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org