You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/15 03:12:17 UTC
[PR] fix(core): resolve Java timestamp deletion vector paths [paimon-rust]
JingsongLi opened a new pull request, #833:
URL: https://github.com/apache/paimon-rust/pull/833
### Purpose
Resolve Java-written bucket-local deletion vectors in timestamp partitions. Java uses `.000` for whole-second non-legacy `TIMESTAMP(3)` values and groups legacy timestamp fractions into 3, 6, or 9 digits. Rust currently computes different directories and can miss the DV or select a stale `table/index` copy. This was reproduced while fixing the review in apache/paimon#9825.
### Brief change log
- Add an internal Java-compatible timestamp partition path calculation for DV reads, including declared precision and legacy fractional widths.
- Preserve explicit DV paths. When Java's exact bucket-local file exists, use it; otherwise retain the existing Rust bucket and legacy table-index resolution.
- Keep the existing data-file and index directory calculation unchanged. A global format change would move the expected location of old Rust files and affect COW file identity; this fix only resolves DV read paths.
### Tests
- Partition utilities: 32 tests passed.
- Index-file path resolution: 14 tests passed, including the table-level resolver with mixed Java/Rust/table-index DVs, same-name decoys, missing files, and authoritative explicit paths.
- New Java timestamp formatting regressions failed before the fix and pass afterward. Default Rust partition path assertions confirm existing directory spellings remain unchanged.
- `cargo fmt --all -- --check` and `cargo clippy --locked -p paimon --lib --tests -- -D warnings` passed.
- Using a wheel rebuilt from this final change, PyPaimon passed 259 related tests (84 native plans), including 91 DV-path cases covering Java layouts, repeated deletes, historical reads, and authoritative path precedence.
### API and Format
No public API or serialization changes. No directory migration is needed for existing Rust tables.
### Documentation
Clarified timestamp formatting and DV path resolution comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): align timestamp partition paths with Java [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #833:
URL: https://github.com/apache/paimon-rust/pull/833#discussion_r4011713398
##########
crates/paimon/src/spec/partition_utils.rs:
##########
@@ -460,26 +461,28 @@ fn format_timestamp_legacy(dt: NaiveDateTime) -> String {
let mut result = format!("{date_hour_min}:{sec:02}");
if nano > 0 {
let frac = format!("{nano:09}");
- let trimmed = frac.trim_end_matches('0');
+ let digits = if nano.is_multiple_of(1_000_000) {
+ 3
+ } else if nano.is_multiple_of(1_000) {
+ 6
+ } else {
+ 9
+ };
result.push('.');
- result.push_str(trimmed);
+ result.push_str(&frac[..digits]);
}
result
}
/// Format a timestamp using non-legacy `DateTimeUtils.formatTimestamp()` semantics.
///
/// Always uses space separator: `yyyy-MM-dd HH:mm:ss[.fraction]`.
-/// Fraction: pad nano to 9 digits, strip trailing zeros down to at most `precision` digits.
+/// Fraction: pad nano to 9 digits, strip trailing zeros while keeping at least `precision` digits.
fn format_timestamp_non_legacy(dt: NaiveDateTime, precision: u32) -> String {
let nano = dt.nanosecond();
let ymdhms = dt.format("%Y-%m-%d %H:%M:%S").to_string();
- if precision == 0 || nano == 0 {
- return ymdhms;
- }
-
- // Pad nano to 9 digits, then strip trailing zeros but keep at least up to `precision` digits.
+ // Whole seconds keep the declared precision; nonzero fractions are never truncated.
Review Comment:
[P1] Preserve reads of timestamp tables written by the released 0.3.0
This shared formatter changes existing-file lookup as well as new writes. The format note acknowledges old Rust directories, but these spellings are also produced by the published `pypaimon-rust==0.3.0`, not just development builds. I installed that wheel and committed two `TIMESTAMP(3)`-partitioned tables: non-legacy `2026-09-15 12:00:00`, and legacy `2026-09-15 12:00:00.100`. Both SQL reads succeed with 0.3.0. Reading the same unchanged warehouses using a wheel built from this head fails with filesystem `NotFound`: the first file exists under `ts=2026-09-15 12%3A00%3A00/` but is sought under `...00.000/`; the second exists under `ts=2026-09-15T12%3A00%3A00.1/` but is sought under `...00.100/`.
The committed metadata does not record an explicit path for these local data files, so an upgrade cannot recover their original locations from the new formatter alone. Please keep Java-canonical paths for new writes while adding a read-side fallback for the old Rust spelling when the canonical file is absent, preserving explicit-path precedence. Add an upgrade test that writes with the released wheel and reads with the new one; fresh-table round trips do not exercise this failure. The same compatibility requirement applies to the legacy fractional-width change above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): align timestamp partition paths with Java [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #833:
URL: https://github.com/apache/paimon-rust/pull/833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): align timestamp partition paths with Java [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #833:
URL: https://github.com/apache/paimon-rust/pull/833#discussion_r4011720503
##########
crates/paimon/src/spec/partition_utils.rs:
##########
@@ -460,26 +461,28 @@ fn format_timestamp_legacy(dt: NaiveDateTime) -> String {
let mut result = format!("{date_hour_min}:{sec:02}");
if nano > 0 {
let frac = format!("{nano:09}");
- let trimmed = frac.trim_end_matches('0');
+ let digits = if nano.is_multiple_of(1_000_000) {
+ 3
+ } else if nano.is_multiple_of(1_000) {
+ 6
+ } else {
+ 9
+ };
result.push('.');
- result.push_str(trimmed);
+ result.push_str(&frac[..digits]);
}
result
}
/// Format a timestamp using non-legacy `DateTimeUtils.formatTimestamp()` semantics.
///
/// Always uses space separator: `yyyy-MM-dd HH:mm:ss[.fraction]`.
-/// Fraction: pad nano to 9 digits, strip trailing zeros down to at most `precision` digits.
+/// Fraction: pad nano to 9 digits, strip trailing zeros while keeping at least `precision` digits.
fn format_timestamp_non_legacy(dt: NaiveDateTime, precision: u32) -> String {
let nano = dt.nanosecond();
let ymdhms = dt.format("%Y-%m-%d %H:%M:%S").to_string();
- if precision == 0 || nano == 0 {
- return ymdhms;
- }
-
- // Pad nano to 9 digits, then strip trailing zeros but keep at least up to `precision` digits.
+ // Whole seconds keep the declared precision; nonzero fractions are never truncated.
Review Comment:
With the requirement clarified, Java is the authoritative format and compatibility with the incorrect directory spellings written by Rust 0.3.0 is intentionally out of scope. I am withdrawing this blocking request; no fallback for those spellings is needed. The Java parity and new-write/read validations passed, and I have no other blocking findings at 5c5acff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org