You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/14 13:13:47 UTC
[PR] fix(python): align native scan planning with PyPaimon [paimon-rust]
JingsongLi opened a new pull request, #831:
URL: https://github.com/apache/paimon-rust/pull/831
### Purpose
Related: #413 (follow-up to the Python scan-planning bindings).
Make native scan plans usable by PyPaimon for incremental and distributed reads while preserving snapshot identity, branch isolation, deletion vectors, and primary-key merge groups. For example, versions of the same primary key in two APPEND snapshots must be planned together; planning each commit separately can return both versions.
### Brief change log
- Add combined APPEND-delta planning for an explicit snapshot range, using the ending snapshot for metadata and deletion vectors.
- Add Data Evolution row-position slices and shards before group-statistics pruning, projection, and deletion-vector filtering. Preserve group order when a later limit selects rows.
- Preserve snapshot IDs for empty plans, expose raw scan traces and runtime capability markers, and resolve branch-qualified table handles and their metadata managers.
- Decode deletion-vector arrays using their actual Avro writer schema and read legacy Python bucket-local index references from `table/index` when the canonical path is absent.
- Accept Python `bytes`/`bytearray` binary literals and keep overlapping composite floating-point key ranges together, including signed zero and NaN bounds.
### Tests
Passed locally:
- Rust table-scan tests: 100.
- Rust incremental batch-scan tests: 31.
- Rust row-position selection and index-path tests: 5 and 13.
- Python binding read/table tests against a newly built wheel: 89.
- `cargo check -p pypaimon_rust`, `cargo fmt --all -- --check`, and `git diff --check`.
The wheel was also validated with companion PyPaimon changes in a separate checkout: 193 planner-parity regressions and 196 reader regressions passed, including ORC. Those companion Python changes are outside this PR. The full workspace and Spark-warehouse integration suites were not run locally.
### API and Format
Adds `Plan.snapshot_id()`, `Table.branch()`, `TableScan.plan_with_trace()`, `ReadBuilder.new_incremental_scan(start, end)`, DE row-position selection methods, and `planning_capabilities()` to the binding, plus the corresponding combined-scan APIs in the core.
Existing per-commit `IncrementalScan.plan()` behavior is preserved. The newly added row-position configuration requires combined planning. Split/storage format versions are unchanged.
NaN key bounds use conservative overlapping groups. Legacy bucket-local DV resolution can add filesystem existence checks; explicit external paths and existing canonical files retain priority. Complete parity for all PyPaimon modes remains follow-up work.
### Documentation
Updated `docs/src/python-binding.md` and the Python type stubs with the new APIs, range semantics, and compatibility guarantees.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(python): align native scan planning with PyPaimon [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #831:
URL: https://github.com/apache/paimon-rust/pull/831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(python): align native scan planning with PyPaimon [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #831:
URL: https://github.com/apache/paimon-rust/pull/831#discussion_r4006286015
##########
crates/paimon/src/table/table_scan.rs:
##########
@@ -1827,11 +1949,26 @@ impl<'a> PaimonTableScan<'a> {
let open_file_cost = core_options.source_split_open_file_cost();
let partition_keys = self.table.schema().partition_keys();
+ // Assign row positions using the full candidate file ranges, before
+ // group stats, projection or DVs change visible rows. Intersect explicit
+ // and index-selected ranges only after assigning the positional range.
+ let effective_row_ranges = if let Some(selection) = self.row_position_selection {
+ Some(selection.select(&entries, effective_row_ranges.as_deref())?)
Review Comment:
[P1] Assign row positions before row-range pruning
`entries` is already pruned by explicit/global-index row ranges here: `plan_snapshot_from_lists` passes `manifest_row_ranges` into `plan_manifest_entries_with_trace`, which applies `retain_manifest_row_ranges` and `retain_manifest_entry_row_ranges`. If that removes a preceding file, `selection.select` renumbers the remaining rows, so intersecting the ranges afterward cannot preserve the requested positions.
For example, persist a DE snapshot with two three-row files covering row IDs `[0,2]` and `[3,5]`, then run:
```python
rb.with_row_ranges([(4, 4)])
plan = rb.new_scan().with_row_position_slice(3, 6).plan()
```
The positional slice should select IDs `[3,5]`, and its intersection with `[4,4]` should retain row 4. Instead, early pruning removes the first file, the remaining three rows are assigned positions `[0,3)`, and the returned plan has no splits. I reproduced this through the public Rust `new_scan().plan()` path against this head; the expected `[RowRange { from: 4, to: 4 }]` was actually `[]`.
Please preserve the complete candidate position space when positional selection is enabled, and only then intersect explicit/index-selected ranges. Disabling the early row-range pruning for positional scans made the reproducer and the existing row-position tests pass locally. Add multi-file coverage where the explicit range removes an entire preceding file, including combined incremental scans and shard selection; the current intersection tests do not exercise that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org