You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/03 15:24:48 UTC

Re: [PR] feat: fuse the typed Dataset map sandwich into a Comet projection [datafusion-comet]

andygrove commented on PR #5714:
URL: https://github.com/apache/datafusion-comet/pull/5714#issuecomment-5970550332

   Closing this in favor of #6564.
   
   Since #6542, the codegen dispatcher only calls into Spark's own classes, so the fused tree this PR builds now falls back to Spark. I confirmed it by merging main locally: 10 of the 18 tests in `CometTypedDatasetSuite` fail with `invoke calls scala.Function1, which is not part of Spark`. Getting it working again would take an exemption to #6542's policy plus special cases for decimal serializers.
   
   #6564 takes a simpler route. It leaves the typed operation on Spark and converts the output of `SerializeFromObjectExec` to Arrow, so the operators above it can run natively. Measured with this PR's benchmark, it got 93-100% of the fused speedup wherever either approach paid off, and it covers `flatMap`, `mapPartitions`, `mapGroups` and `cogroup` as well as `map`. A trimmed version of the benchmark moved over with it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org