[PR] fix(datafusion): keep format table partition columns out of decoder filters [paimon-rust]

["sundapeng (via GitHub)" <gi...@apache.org>]

sundapeng opened a new pull request, #811:
URL: https://github.com/apache/paimon-rust/pull/811

   ### Purpose
   
   A format table keeps its partition values in directory names only; its data files do not hold the partition columns. A filter on a partition column that the scan cannot turn into a partition predicate, such as a bare boolean column after DataFusion simplifies `active = true` to `active`, or `upper(dt) = 'B'`, is also installed as a Parquet decoder row filter. The decoder evaluates it against the columns of the data file, where the partition column is missing, so every row is dropped before the runtime filter sees the batch with the partition columns filled in. The query returns no rows and no error.
   
   This was found while working on #591 and is split out of it, since it affects every partitioned format table, not only catalog-managed ones.
   
   ### Brief change log
   
   - `PaimonTableScan` no longer installs a conjunct as a decoder filter when it reads a partition column of a format table. Such conjuncts stay with the runtime filters, which run on batches with the partition columns filled in.
   - Paimon tables are unaffected, since their data files carry the partition columns.
   
   ### Tests
   
   - New `crates/integrations/datafusion/tests/format_table_partition_filters.rs`: a format table partitioned by `(dt, active)` on a filesystem catalog, queried with `active`, `active = true`, `NOT active`, `upper(dt) = 'B'` and `concat(dt, '-') = 'a-'`, asserting the exact rows. On main every one of those predicates returns no rows, while the control `dt IN ('a', 'b')` returns both; with this change all of them pass.
   - The test catalog uses a `file:/...` warehouse. With the `file://...` form used by the shared test helper, a scan that lists from the table root cannot relate files to their partition directories and returns nothing, a separate issue already noted in `format_table_statistics.rs`.
   
   ### API and Format
   
   No API or storage format change.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(datafusion): keep format table partition columns out of decoder filters [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #811:
URL: https://github.com/apache/paimon-rust/pull/811


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org