[jira] [Created] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

["Wen Yuen Pang (Jira)" <ji...@apache.org>]

Wen Yuen Pang created SPARK-44934:
-------------------------------------

             Summary: PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes
                 Key: SPARK-44934
                 URL: https://issues.apache.org/jira/browse/SPARK-44934
             Project: Spark
          Issue Type: Bug
          Components: Optimizer
    Affects Versions: 3.4.1
            Reporter: Wen Yuen Pang


When running the query

```
with cte as (
 select c1, c1, c2, c3 from t where random() > 0
)
select cte.c1, cte2.c1, cte.c2, cte2.c3 from
 (select c1, c2 from cte) cte
 inner join
 (select c1, c3 from cte) cte2
 on cte.c1 = cte2.c1
```
 
The query fails with the error
```
org.apache.spark.scheduler.DAGScheduler: Failed to update accumulator 9523 (Unknown class) for task 1

org.apache.spark.SparkException: attempted to access non-existent accumulator 9523

```

Further investigation shows that the rule `PushdownPredicatesAndPruneColumnsForCTEDef` creates an invalid plan when the output of a CTE contains duplicate expression IDs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org