[GitHub] [iceberg] singhpk234 commented on issue #6044: Column pruning/projection is not happening in correlated queries (e.g Q94, Q16)

[GitBox <gi...@apache.org>]

singhpk234 commented on issue #6044:
URL: https://github.com/apache/iceberg/issues/6044#issuecomment-1295643951

   Can you please also add spark version you are observing this ? 
   
   I tried this with a sample UT below with spark 3.3 I could see schema pruning happening : 
   
   UT : 
   
   ```java
     @Test
     public void testSchemaPruningInCorrelatedSubQueries() {
       String tbl1 = tableName("tbl_1");
       sql("CREATE TABLE %s (id INT, name STRING) USING iceberg", tbl1);
       sql("INSERT INTO TABLE %s VALUES (1, 'PK')", tbl1);
       sql("INSERT INTO TABLE %s VALUES (2, 'RK')", tbl1);
   
       String tbl2 = tableName("tbl_2");
       sql("CREATE TABLE %s (id INT, name STRING) USING iceberg", tbl2);
       sql("INSERT INTO TABLE %s VALUES (1, 'PK')", tbl2);
       sql("INSERT INTO TABLE %s VALUES (2, 'RK')", tbl2);
   
       spark.sql(String.format("SELECT id FROM %s a WHERE EXISTS (SELECT * FROM %s b WHERE a.id = b.id)", tbl1, tbl2)).queryExecution().executedPlan();
   
       sql("DROP TABLE IF EXISTS %s", tbl1);
       sql("DROP TABLE IF EXISTS %s", tbl2);
     }
   ```
   
   Spark Plan : 
   ```
   AdaptiveSparkPlan isFinalPlan=false
   +- BroadcastHashJoin [id#201], [id#203], LeftSemi, BuildRight, false
      :- BatchScan[id#201] spark_catalog.default.tbl_1 [filters=] RuntimeFilters: []
      +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#316]
         +- Project [id#203]
            +- BatchScan[id#203, name#204] spark_catalog.default.tbl_2 [filters=] RuntimeFilters: []
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org