You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink Jira Bot (Jira)" <ji...@apache.org> on 2021/11/27 23:08:00 UTC
[jira] [Updated] (FLINK-21350) ParquetInputFormat incorrectly interprets timestamps encoded in microseconds as timestamps encoded in milliseconds
[ https://issues.apache.org/jira/browse/FLINK-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flink Jira Bot updated FLINK-21350:
-----------------------------------
Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.
> ParquetInputFormat incorrectly interprets timestamps encoded in microseconds as timestamps encoded in milliseconds
> ------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-21350
> URL: https://issues.apache.org/jira/browse/FLINK-21350
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.12.0, 1.12.1
> Reporter: Jeffrey Charles
> Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> Given a parquet file with a schema that has a field with a physical type of INT64 and a logical type of TIMESTAMP_MICROS, all of the ParquetInputFormat sub-classes deserialize the timestamp as tens of thousands of years in the future.
> Looking at the code in [https://github.com/apache/flink/blob/release-1.12.1/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/utils/RowConverter.java#L326,] it looks to me like the row converter is interpreting the field value as if it contained milliseconds and not microseconds. Specifically both millisecond and microsecond processing share the same code path to instantiate a java.sql.timestamp which takes a millisecond value in its constructor and the microsecond case statement is passing it a value in microseconds. I tested a change locally where I divide the value by 1000 in the microseconds case statement and that results in a timestamp with the expected value.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)