You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:16:17 UTC

[I] TIMESTAMP_MICROS handling [hudi]

hudi-bot opened a new issue, #17222:
URL: https://github.com/apache/hudi/issues/17222

   Hi Guys!
    
   I am not able to use timestamp micro columns save with HUDI. 
   I would like to save it keeping microsec granularity, but it only keeps milisec.
    
   I have set this:
   --conf spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS \
   and also this in the hoodie:
   "hoodie.parquet.outputtimestamptype": "TIMESTAMP_MICROS",
   but when I read it back (with pyspark, load api), it's only millisecond precision and unfortunately, I need the microsec in some case, because with this I run into a Schrödinger's cat situation   !https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f604.png!
   So an entity has more than one states in the same time !https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f604.png!Can someone enlighten me what should I do?
    
   Before the save, everything is fine! ("ts" column)
   
   Darvi
   SLACK Thread: [https://apache-hudi.slack.com/archives/C4D716NPQ/p1652347742173779]
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-4091
   - Type: Sub-task
   - Parent: https://issues.apache.org/jira/browse/HUDI-9113
   - Affects version(s):
     - 0.10.1
   - Fix version(s):
     - 1.1.0
   - Attachment(s):
     - 12/May/22 11:37;Darvi77;b97b9e55-58a4-417b-b71c-f6b2d3860da0-0_0-26-1663_20220512111505310.parquet;https://issues.apache.org/jira/secure/attachment/13043566/b97b9e55-58a4-417b-b71c-f6b2d3860da0-0_0-26-1663_20220512111505310.parquet
     - 12/May/22 11:38;Darvi77;before-save.png;https://issues.apache.org/jira/secure/attachment/13043565/before-save.png
     - 12/May/22 11:37;Darvi77;example-code.txt;https://issues.apache.org/jira/secure/attachment/13043567/example-code.txt


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] TIMESTAMP_MICROS handling [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17222:
URL: https://github.com/apache/hudi/issues/17222#issuecomment-5366110156

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-4091).
   
   **Findings: needs a reproduction against a current build before it can be dispositioned.**
   
   The report is concrete: timestamps written with spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS and hoodie.parquet.outputtimestamptype=TIMESTAMP_MICROS come back at millisecond precision on read. Reproducing it looks tractable - a small write-then-read round trip with a microsecond-precision timestamp is enough; no cluster needed.
   
   I have not called this from the source. The surrounding code has been rewritten substantially since this was filed - table versions 8 and 9, the `BufferedRecordMerger` family replacing payload-based merging, the file-group-reader migration, the indexer abstraction in #18348 / #18372, and the `hudi-common` package moves in #19193 / #19195. Inferring either "still broken" or "silently fixed" through that much churn would be a guess.
   
   There is also no commit anywhere in the tree referencing HUDI-4091, so if it was fixed it was incidental rather than deliberate - another reason to verify rather than assume.
   
   This is one of four timestamp-precision reports in the backlog and they should be triaged as a set rather than individually: #17302 (HUDI-8592, `hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS` ignored, parquet always micros), #17247 (HUDI-7306, CustomKeyGenerator dropping the microseconds part), #17329 (HUDI-8677, Hive truncating micros to millis), and this one. They may share a single root cause in how the output timestamp type is threaded from Hudi options into the parquet writer - and note #17302 reports the opposite symptom to this ticket, which is itself informative.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org