You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2026/04/27 07:13:19 UTC
[PR] [python] Fix FileStoreCommit overwrite use full table fields for sanity check [paimon]
SteNicholas opened a new pull request, #7711:
URL: https://github.com/apache/paimon/pull/7711
### Purpose
Fix `FileStoreCommit` overwrite use full table fields for sanity check.
### Tests
- `TestFileStoreCommit: test_overwrite_accepts_matching_partition`
- `TestFileStoreCommit: test_overwrite_rejects_mismatched_partition`
- `TestFileStoreCommit: test_overwrite_predicate_index_matches_schema_scope`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix FileStoreCommit overwrite use full table fields for sanity check [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7711:
URL: https://github.com/apache/paimon/pull/7711#discussion_r3145830757
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -170,15 +170,23 @@ def overwrite(self, overwrite_partition, commit_messages: List[CommitMessage], c
partition_filter = None
# sanity check, all changes must be done within the given partition, meanwhile build a partition filter
if len(overwrite_partition) > 0:
- predicate_builder = PredicateBuilder(self.table.partition_keys_fields)
- sub_predicates = []
+ # Build two predicates from overwrite_partition in one pass:
+ # - check_filter: indexed by partition keys only, used for the sanity check
+ # below because msg.partition only contains partition key values.
+ # - partition_filter: indexed by full table schema, used by FileScanner's
Review Comment:
Can we just let `FileScanner` supports partition_predicate？
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -170,15 +170,23 @@ def overwrite(self, overwrite_partition, commit_messages: List[CommitMessage], c
partition_filter = None
# sanity check, all changes must be done within the given partition, meanwhile build a partition filter
if len(overwrite_partition) > 0:
- predicate_builder = PredicateBuilder(self.table.partition_keys_fields)
- sub_predicates = []
+ # Build two predicates from overwrite_partition in one pass:
+ # - check_filter: indexed by partition keys only, used for the sanity check
+ # below because msg.partition only contains partition key values.
+ # - partition_filter: indexed by full table schema, used by FileScanner's
Review Comment:
Can we just let `FileScanner` supports partition_predicate just like Java and Rust？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support FileScanner partition_predicate to fix FileStoreCommit overwrite for sanity check [paimon]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #7711:
URL: https://github.com/apache/paimon/pull/7711#issuecomment-4334003224
Ping @JingsongLi. Addressed above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix FileStoreCommit overwrite use full table fields for sanity check [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #7711:
URL: https://github.com/apache/paimon/pull/7711#issuecomment-4325047102
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix FileStoreCommit overwrite use full table fields for sanity check [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7711:
URL: https://github.com/apache/paimon/pull/7711#discussion_r3145830757
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -170,15 +170,23 @@ def overwrite(self, overwrite_partition, commit_messages: List[CommitMessage], c
partition_filter = None
# sanity check, all changes must be done within the given partition, meanwhile build a partition filter
if len(overwrite_partition) > 0:
- predicate_builder = PredicateBuilder(self.table.partition_keys_fields)
- sub_predicates = []
+ # Build two predicates from overwrite_partition in one pass:
+ # - check_filter: indexed by partition keys only, used for the sanity check
+ # below because msg.partition only contains partition key values.
+ # - partition_filter: indexed by full table schema, used by FileScanner's
Review Comment:
Can we just let `FileScanner` supports partition_predicate just like Java and Rust? In the future, if there are too many partitions, we can also carry out targeted optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support FileScanner partition_predicate to fix FileStoreCommit overwrite for sanity check [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7711:
URL: https://github.com/apache/paimon/pull/7711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support FileScanner partition_predicate to fix FileStoreCommit overwrite for sanity check [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #7711:
URL: https://github.com/apache/paimon/pull/7711#issuecomment-4333611945
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org