You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/28 12:50:41 UTC
[PR] fix(variant): store shredded timestamps as microseconds [paimon-rust]
jackylee-ch opened a new pull request, #983:
URL: https://github.com/apache/paimon-rust/pull/983
The Variant binary stores timestamps as microseconds (encoding type codes 12 and 13 are `TIMESTAMP(MICROS)` / `TIMESTAMP_NTZ(MICROS)`), and the shredding spec requires the shredded `typed_value` column to use microseconds. A configured shredding schema, however, had its declared type copied through verbatim, so declaring `TIMESTAMP(3)` produced a millisecond-typed `typed_value` column that still held the microsecond count.
Rust round-trips this losslessly because it ignores the unit on both sides, but the file violates the spec and any precision-aware reader (Spark, DuckDB, Arrow, Java `Timestamp.fromMicros`) reads it 1000x off.
Pin the shredded timestamp `typed_value` to microseconds so the column is annotated consistently; other scalars are unchanged. A test asserts this holds even when the configured schema declares another precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(variant): store shredded timestamps as microseconds [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #983:
URL: https://github.com/apache/paimon-rust/pull/983#issuecomment-5870477840
Superseded by #882, which fixes the same shredded-timestamp unit issue and is already under review. Closing to consolidate; the microsecond-pinning approach will be folded into #882.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(variant): store shredded timestamps as microseconds [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch closed pull request #983: fix(variant): store shredded timestamps as microseconds
URL: https://github.com/apache/paimon-rust/pull/983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(variant): store shredded timestamps as microseconds [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #983:
URL: https://github.com/apache/paimon-rust/pull/983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(variant): store shredded timestamps as microseconds [paimon-rust]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #983:
URL: https://github.com/apache/paimon-rust/pull/983#issuecomment-5882665156
Reopening. Re-reading the Parquet Variant shredding spec, a shredded timestamp `typed_value` must be MICROS or NANOS (VariantShredding.md: "Shredded values must use the following Parquet types", which lists only `TIMESTAMP(MICROS)` / `TIMESTAMP(NANOS)`). Pinning the shredded timestamp to microseconds — as this PR does — is the spec-conformant fix and matches the microsecond values Paimon-Java's inferred shredding schema writes. Continuing here; #882 (which rescaled to the declared precision, yielding a MILLIS/SECOND leaf) is being closed in favor of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org