You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anuragmantri (via GitHub)" <gi...@apache.org> on 2026/10/22 19:38:06 UTC

[PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

anuragmantri opened a new pull request, #18615:
URL: https://github.com/apache/iceberg/pull/18615

   Follow-up to #17956, which returns every opaque partition predicate to Spark for re-evaluation. After this PR, a predicate is returned only when some spec in the table cannot evaluate it, using the same per-spec check the scan uses to prune tasks. 
   
   This PR adds two guards to keep results correct. 
   - The changelog builder still returns these predicates because its scan does not evaluate them. 
   - Aggregate pushdown is skipped when partition predicates are present, since it ignores them and `count(*)` returns the unfiltered count otherwise. 
    
   The check covers all table specs rather than only scanned ones, which is conservative. Narrowing it is possible as a follow-up.
   
   Test plan: 
   - builder tests cover the dropped, kept and spec-evolved cases, including a field whose ordinal differs across specs, plus the changelog builder. 
   - SQL tests cover count(*) and results on an evolved spec.
   
   ---
   AI Disclosure
    - Model: Claude Sonnet 5.5
    - Platform/Tool: Claude Code
    - Human Oversight: fully reviewed by me.
    - Prompt Summary: Implement the TODO in BaseSparkScanBuilder.pushPredicates left by #17956


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6353682595

   Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6352518701

   @szehon-ho. For sure, since you have all the context, I will close this. You can take over. Fell free to take this code if it helps you. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6350316915

   Hi @anuragmantri this is purposeful as there's a correctness issue in Spark 4.2 if there's an exception in the connector the row passes.  Ill fix it once we release the new spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6352200563

   Thanks for the context @szehon-ho, that makes sense. I reproduced it to confirm. The following test fails on this PR.
   
   ```Java
   @TestTemplate
   void partitionPredicateErrorIsNotSwallowed() {
     sql("CREATE TABLE %s (id INT, dep STRING) USING iceberg PARTITIONED BY (dep)", tableName);
     configurePlanningMode(planningMode);
   
     setSmallSplitSize();
     sql("INSERT INTO %s VALUES (1, 'd1')", tableName);
     sql("INSERT INTO %s VALUES (2, 'd2')", tableName);
     sql("INSERT INTO %s VALUES (3, 'd3')", tableName);
   
     spark
         .udf()
         .register(
             "fails_on_d3",
             (String value) -> {
               if ("d3".equals(value)) {
                 throw new IllegalStateException("Cannot evaluate " + value);
               }
   
               return "d2".equals(value);
             },
             DataTypes.BooleanType);
   
     assertThatThrownBy(() -> sql("SELECT * FROM %s WHERE fails_on_d3(dep)", tableName))
         .as("Spark must evaluate the predicate on rows of the partition where pruning failed")
         .hasMessageContaining("fails_on_d3");
   }
   ```
   
   I'm moving this PR to draft until the Spark fix is released. After that I'll rebase and add the test here as a regression test. Do you have the Spark issue number I can link here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6349627063

   @szehon-ho @huaxingao - Could you take a look? thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri closed pull request #18615: Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates
URL: https://github.com/apache/iceberg/pull/18615


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Skip Spark re-evaluation of fully evaluated partition predicates [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6352403282

   Thanks for confirming.  Yea I was planning to work on it, if its ok?  Im doing the spark 4.2.1 rc now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org