You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/14 06:13:45 UTC
[PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
jackylee-ch opened a new pull request, #825:
URL: https://github.com/apache/paimon-rust/pull/825
`decode_nullable_dv_ranges` read the `_DELETIONS_VECTORS_RANGES` item record
positionally and always consumed a `_CARDINALITY` union index. Java added that
field in 1.0.0 (#4699); 0.8.0-0.9.x declared the item as `RowType.of(STRING,
INT, INT)`, so on an older file the decoder steals the next item's bytes and
`IndexManifest::read_from_bytes` fails with `avro cursor: negative bytes
length: -49`. The serde reader reads the same bytes fine, and the read is also
on the commit path (`read_prev_index_entries`), not just scans.
**Fix**: walk the writer's own nested field list, as
`decode_nullable_binary_table_stats` does, peeling `array<["null", record]>` to
reach the item, since array items keep their union. An item without `f0` is
rejected rather than defaulted, `f0` being the map key.
An index manifest is rewritten only by a commit that changes an index file, so
such files are still live.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #825:
URL: https://github.com/apache/paimon-rust/pull/825#issuecomment-5661126273
@jackylee-ch Why is it still necessary to maintain compatibility with formats from a few years ago?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #825:
URL: https://github.com/apache/paimon-rust/pull/825#issuecomment-5676152065
Closing this — #831 already covers it.
That change walks the writer's own field list for the deletion-vector item
record, which is what this PR did, and goes further: it accepts a plain
(non-union) item as PyPaimon writes it, where this PR would have rejected the
schema outright, and it tolerates reordered and unknown fields. Its
`test_deletion_vectors_follow_writer_item_schema` already covers the two-entry,
two-item shape across both nullable items and a missing `_CARDINALITY`.
Your pushback was right: the version-compatibility framing was the weak part of
this PR. The mechanism was the substance, and it has landed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch closed pull request #825: fix(spec): decode pre-1.0 deletion-vector metas via the writer schema
URL: https://github.com/apache/paimon-rust/pull/825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #825:
URL: https://github.com/apache/paimon-rust/pull/825#issuecomment-5662025929
Agreed, 0.8/0.9 support on its own would not be worth it — that table is just how I hit it. As I read `IndexManifestEntrySerializer`, this record's field list is deliberately not fixed: "Do not change when adding nullable fields. Old manifest readers skip unknown fields." Java resolves the record by name against the OCF's writer schema, so an item without `_CARDINALITY` reads as null; ours reads that union index by position, so the gap is on our side, not Java's. `decode_nullable_global_index` guards `_SOURCE_META` the same way (#515). Happy to drop the version framing from title and body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #825:
URL: https://github.com/apache/paimon-rust/pull/825#issuecomment-5660699843
Not directly from a production issue. I found this while testing compatibility with tables written by older Paimon versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spec): decode pre-1.0 deletion-vector metas via the writer schema [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #825:
URL: https://github.com/apache/paimon-rust/pull/825#issuecomment-5660562348
Does this requirement come from a real production environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org