[jira] [Assigned] (SPARK-39362) Datasource v2 scan not pruned in tpcds q10

["Apache Spark (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-39362:
------------------------------------

    Assignee:     (was: Apache Spark)

> Datasource v2 scan not pruned in tpcds q10
> ------------------------------------------
>
>                 Key: SPARK-39362
>                 URL: https://issues.apache.org/jira/browse/SPARK-39362
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.2.1
>         Environment: Spark 3.2 + Iceberg 3.2
>            Reporter: Zhen Wang
>            Priority: Major
>
> {code:java}
>     createAndInitTable("id INT, dep STRING");
>     createAndInitTable(
>         wideTableName, "id INT, c1 STRING, c2 STRING, c3 STRING, c4 STRING, c5 STRING", null);
>     String q1 = String.format("select dep from %s t1 where id > 1 and exists " +
>         "(select * from %s t2  where t1.id = t2.id)", tableName, wideTableName);
>     Assert.assertEquals("should prune scan", 1, scanCols);
> {code}
> it looks like select * in subqueries is not fully projected when v2ScanRelationPushdown is applied. after it is later projected the scan wont be changed after that. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org