Hemanth Boyina created SPARK-60439:
--------------------------------------
Summary: Support DataSource V2 pushdown for aggregates with FILTER clauses
Key: SPARK-60439
URL: https://issues.apache.org/jira/browse/SPARK-60439
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
Spark supports aggregate pushdown, but aggregates with a FILTER clause are currently evaluated in Spark rather than the data source.
For example:
SELECT
SUM(amount) FILTER (WHERE status = 'paid'),
COUNT(*) FILTER (WHERE status = 'failed'),
COUNT(*)
FROM sales;
Allow supported data sources to compute these measures by translating each filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces input rows transferred to Spark while preserving each measure’s condition.
Initial support covers deterministic filters on non-distinct MIN, MAX, SUM, AVG, and single-input COUNT, including COUNT(*). Require complete aggregate pushdown and retain Spark execution for unsupported cases. No new public API is needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org