You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2026/08/20 09:47:05 UTC

[PR] [spark] Fix timestamp precision in key dynamic writes [paimon]

Zouxxyy opened a new pull request, #9322:
URL: https://github.com/apache/paimon/pull/9322

   ### Purpose
   
   Fix Spark key-dynamic writes for Paimon tables containing timestamps whose precision cannot be represented in Spark schemas.
   
   The bootstrap path serializes rows with the table's original `RowType`, while the global-index output was decoded with a `RowType` reconstructed from the Spark schema. Spark loses the Paimon timestamp precision, so a `TIMESTAMP(3)` binary row was decoded as `TIMESTAMP(6)` and failed.
   
   Decode global-index data rows with `fileStoreTable.rowType()` and append the Spark row-kind and bucket columns separately.
   
   ### Tests
   
   - Added a cross-partition dynamic-bucket regression test using `TIMESTAMP(3)` and deletion vectors.
   - Spark 3 `DynamicBucketTableTest`: 8 tests passed.
   - Spark 4 shared sources compiled successfully.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [spark] Fix timestamp precision in key dynamic writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9322:
URL: https://github.com/apache/paimon/pull/9322


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [spark] Fix timestamp precision in key dynamic writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9322:
URL: https://github.com/apache/paimon/pull/9322#issuecomment-5363707863

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org