You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/24 13:13:26 UTC
[PR] [python] Apply filters when reading system tables [paimon]
jackylee-ch opened a new pull request, #10164:
URL: https://github.com/apache/paimon/pull/10164
### Purpose
`SystemReadBuilder.with_filter` stores and forwards a predicate, but `SystemTableRead._materialise` raised `NotImplementedError` for **any** predicate. So `with_filter(...)` on a system table (`db.t$snapshots`, `db.t$files`, ...) always failed, and layers that push filters down to pypaimon (Daft / Ray) hit this on every system table.
This applies the predicate to the materialised PyArrow table for the methods that convert to a safe Arrow row filter (comparisons, null checks, `is_in`), reusing `predicate_supports_arrow_filter` from the data-read path. The filter runs before projection, so a predicate may reference columns that are not projected. String-match predicates (`starts_with` / `ends_with` / `contains` / `like`) are not safe as final Arrow row filters and still raise a clear error.
### Tests
`pypaimon/tests/system/system_table_test.py`: added `SystemTableFilterTest` covering equal / greater-than / is-in filtering, filtering on a column that is not in the projection, and the string-match method still raising `NotImplementedError`.
### API and Format
No public API or format change; `with_filter` was already exposed on the system-table read builder.
Written with Claude Code; verification is mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Apply filters when reading system tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10164:
URL: https://github.com/apache/paimon/pull/10164#issuecomment-5831952474
Reviewed the current head end to end. This fixes an existing public `with_filter` path that always failed for system tables, so it has direct value for catalog users and integrations that push down filters. The implementation checks Arrow-safe methods, rejects value comparisons on nested system columns with a clear error, applies the filter before projection, and retains the documented rejection for non-`Predicate` objects and string-match predicates. The two issues raised in the earlier review are addressed at this head.
Verification: all 93 system-table tests passed locally, and `git diff --check` passed. I also seeded a real catalog-backed `db.t$snapshots` table with two snapshots, filtered `snapshot_id = 2` while projecting only `commit_user`, and got exactly `[{"commit_user": "bob"}]`. CI is green across the scheduled Python versions and Native CI. I found no blocking regression. Comparisons on nested fields and string-match filters remain unsupported by design and report that explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Apply filters when reading system tables [paimon]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #10164:
URL: https://github.com/apache/paimon/pull/10164#issuecomment-5825508061
Thanks @Akash3121 — both addressed (rebased onto master):
1. Added an `isinstance(_, Predicate)` guard in `_materialise`, so `with_filter(object())` again raises the documented `NotImplementedError`, not an `AttributeError` (fixes `test_filter_is_not_supported_yet`).
2. Added `_reject_non_scalar_filter_fields`: comparisons / is-in on non-scalar columns (`write_cols`, `dv_ranges`) now raise the documented `NotImplementedError` up front (null checks still pushed down). Added regression tests for an array column and the non-Predicate case; `tests/system` green (93 passed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Apply filters when reading system tables [paimon]
Posted by "Akash3121 (via GitHub)" <gi...@apache.org>.
Akash3121 commented on code in PR #10164:
URL: https://github.com/apache/paimon/pull/10164#discussion_r4096909149
##########
paimon-python/pypaimon/table/system/system_table_read.py:
##########
@@ -95,12 +96,20 @@ def to_duckdb(
return con
def _materialise(self, splits: List[Split]) -> pyarrow.Table:
- if self.predicate is not None:
+ from pypaimon.read.push_down_utils import predicate_supports_arrow_filter
+
+ if self.predicate is not None and not predicate_supports_arrow_filter(
Review Comment:
Please preserve the existing rejection path for unsupported predicate objects before calling this helper. `SystemReadPipelineTest.test_filter_is_not_supported_yet` passes `object()` and expects the public `NotImplementedError` , but `predicate_supports_arrow_filter` immediately dereferences `.method` , so this PR changes the result to an internal `AttributeError` . The focused system-table suite currently fails on this test. An `isinstance(self.predicate, Predicate)` guard (or making the helper return `False` for unknown objects) would retain the previous error contract.
##########
paimon-python/pypaimon/table/system/system_table_read.py:
##########
@@ -110,6 +119,10 @@ def _materialise(self, splits: List[Split]) -> pyarrow.Table:
+ type(split).__name__
)
arrow_table = split.arrow_table()
+ # Filter on the full schema before projection so a predicate may
+ # reference columns that are not part of the projection.
+ if arrow_predicate is not None:
Review Comment:
`predicate_supports_arrow_filter` only checks the predicate method, so it accepts comparisons for every field type. System tables expose array fields such as `$files.write_cols` and `$table_indexes.dv_ranges` , but PyArrow has no equality kernel for list values. For example, `builder.equal("write_cols", ["a"])` reaches this line and raises `ArrowNotImplementedError: Function 'equal' has no kernel matching input types (list<item: string>, list<item: string>)` . Please make the support check field-type/method aware (or provide the existing row-wise fallback) and reject unsupported combinations with the documented `NotImplementedError` . A regression test using an array-valued system-table column would cover this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Apply filters when reading system tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10164:
URL: https://github.com/apache/paimon/pull/10164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org