Re: [PR] fix: fall back to Spark for regr_* aggregates until their merge matches Spark [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove commented on PR #6451:
URL: https://github.com/apache/datafusion-comet/pull/6451#issuecomment-5931283430

   @parthchandra Yes, sharing the key is intended, and it can't be split. Spark plans both `regr_sxx` and `regr_syy` as `RegrReplacement`, so Comet sees one expression class whichever function was called. The compatibility table says so on both rows.
   
   The partition-count assert went away with that Scala test. After @comphead's review it moved to `regr_fallback.sql` in 4935e8dd47. Each query there checks its function's fallback, which is decided at planning time, so it doesn't depend on the layout. The layout only matters for the fixture to reproduce the wrong answer with the serde change reverted. It writes two files with `INSERT ... SELECT ... FROM range(a, b, 1, 1)`, and under the harness's `local[5]` the 4 MB open cost keeps two small files in separate partitions. With the serde reverted, it fails with the issue's wrong `regr_slope`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org