You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/06/14 16:19:18 UTC

[I] Improve performance of TPC-H q19 [datafusion-comet]

andygrove opened a new issue, #572:
URL: https://github.com/apache/datafusion-comet/issues/572

   ### What is the problem the feature request solves?
   
   Comet is currently slower than Spark for query 19.
   
   Some initial observations:
   
   
   - The sort merge join cannot run natively due to https://github.com/apache/datafusion-comet/issues/398
   - The Parquet scan of lineitem seems to take ~10% longer than Spark and 60%+ of the time is spent in native decoding, so perhaps we should add criterion benchmarks for decoding for all types in lineitem and look for optimization opportunities there. I tested both before and after the recent changes to this code and saw no difference.
   - Comet avoids a very expensive C2R on 600 million rows from lineitem because it applies a filter before any C2R, so it is suprising that we are still slower
   - With Comet, there is a really slow C2R on the `part` table where it takes 18 seconds for 48k rows. Spark performs the C2R on 20 million rows and then filters down to 48k and that whole process only takes 3.2 seconds. 
   - Spark coalesces down to 9 partitions and the HashAggregate takes 5.7 seconds and produces 9 rows, but we disable coalesce partitions with Comet and the HashAggregate there takes 11.2 seconds and produces 200 rows. Fixing https://github.com/apache/datafusion-comet/issues/387 would help with this
   
   ### Describe the potential solution
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of TPC-H q19 [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #572:
URL: https://github.com/apache/datafusion-comet/issues/572#issuecomment-2366870966

   Comet is now faster than Spark for this query, and there is no longer a C2R in the Comet plan, so closing this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of TPC-H q19 [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #572: Improve performance of TPC-H q19
URL: https://github.com/apache/datafusion-comet/issues/572


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org