[ https://issues.apache.org/jira/browse/SPARK-60439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Boyina updated SPARK-60439:
-----------------------------------
Description:
Spark supports aggregate pushdown, but aggregates with a FILTER clause are currently evaluated in Spark rather than the data source.
For example:
SELECT
SUM(amount) FILTER (WHERE status = 'paid'),
COUNT(*) FILTER (WHERE status = 'failed'),
COUNT(*)
FROM sales;
Allow supported data sources to compute these measures by translating each filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces input rows transferred to Spark while preserving each measure’s condition.
Initial support covers deterministic filters on non-distinct MIN, MAX, SUM, AVG, and single-input COUNT, including COUNT(*). Require complete aggregate pushdown and retain Spark execution for unsupported cases. No new public API is needed.
was:
Spark supports aggregate pushdown, but aggregates with a FILTER clause are currently evaluated in Spark rather than the data source.
For example:
SELECT
SUM(amount) FILTER (WHERE status = 'paid'),
COUNT(*) FILTER (WHERE status = 'failed'),
COUNT(*)
FROM sales;
Allow supported data sources to compute these measures by translating each filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces input rows transferred to Spark while preserving each measure’s condition.
Initial support covers deterministic filters on non-distinct MIN, MAX, SUM, AVG, and single-input COUNT, including COUNT(*). Require complete aggregate pushdown and retain Spark execution for unsupported cases. No new public API is needed.
> Support DataSource V2 pushdown for aggregates with FILTER clauses
> -----------------------------------------------------------------
>
> Key: SPARK-60439
> URL: https://issues.apache.org/jira/browse/SPARK-60439
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Hemanth Boyina
> Priority: Major
>
> Spark supports aggregate pushdown, but aggregates with a FILTER clause are currently evaluated in Spark rather than the data source.
> For example:
>
> SELECT
> SUM(amount) FILTER (WHERE status = 'paid'),
> COUNT(*) FILTER (WHERE status = 'failed'),
> COUNT(*)
> FROM sales;
>
> Allow supported data sources to compute these measures by translating each filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces input rows transferred to Spark while preserving each measure’s condition.
>
> Initial support covers deterministic filters on non-distinct MIN, MAX, SUM, AVG, and single-input COUNT, including COUNT(*). Require complete aggregate pushdown and retain Spark execution for unsupported cases. No new public API is needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)