[PR] WIP: [spark] Avoid row-id pushdown for non-DE row tracking tables [paimon]

["leaves12138 (via GitHub)" <gi...@apache.org>]

leaves12138 opened a new pull request, #8525:
URL: https://github.com/apache/paimon/pull/8525

   ### Purpose
   
   For a row-tracking table without data evolution, Spark can push predicates on `_ROW_ID` from the `$row_tracking` system table into the Paimon scan. The underlying append-only scan still uses data-file stats for physical columns only, so planning can fail when stats predicates access the row-tracking field index.
   
   This patch keeps row-tracking field predicates as Spark residual filters when row tracking is enabled but data evolution is disabled. Data-evolution row-tracking scans keep the existing pushdown behavior.
   
   ### Changes
   
   - Skip Paimon predicate pushdown for `_ROW_ID` and `_SEQUENCE_NUMBER` predicates on non-DE row-tracking tables.
   - Add a Spark regression test for querying `$row_tracking` with `_ROW_ID = 0`.
   
   ### Tests
   
   - `mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DskipTests compile`
   - `mvn -pl paimon-spark/paimon-spark-common,paimon-spark/paimon-spark-ut -Pspark3 -DskipTests -Dcheckstyle.skip=true -Drat.skip=true spotless:apply`
   - `mvn -pl paimon-spark/paimon-spark-3.5 -am -Pfast-build -Pspark3 -DwildcardSuites=org.apache.paimon.spark.sql.RowTrackingTest -Dtest=none -DfailIfNoTests=false test` (the added test passes; the full suite has existing local CodeGenerator plugin discovery failures)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] WIP: [spark] Avoid row-id pushdown for non-DE row tracking tables [paimon]

["leaves12138 (via GitHub)" <gi...@apache.org>]

leaves12138 closed pull request #8525: WIP: [spark] Avoid row-id pushdown for non-DE row tracking tables
URL: https://github.com/apache/paimon/pull/8525


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org