You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2020/03/17 14:12:00 UTC
[jira] [Resolved] (SPARK-31150) Parsing seconds fraction with
variable length for timestamp
[ https://issues.apache.org/jira/browse/SPARK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-31150.
---------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 27906
[https://github.com/apache/spark/pull/27906]
> Parsing seconds fraction with variable length for timestamp
> -----------------------------------------------------------
>
> Key: SPARK-31150
> URL: https://issues.apache.org/jira/browse/SPARK-31150
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 3.0.0
>
>
> This JIRA is to support parsing timestamp values with variable length second fraction parts.
> e.g. 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]' can parse timestamp with 0~6 digit-length second fraction but fail >=7
> {code:java}
> select to_timestamp(v, 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]') from values
> ('2019-10-06 10:11:12.'),
> ('2019-10-06 10:11:12.0'),
> ('2019-10-06 10:11:12.1'),
> ('2019-10-06 10:11:12.12'),
> ('2019-10-06 10:11:12.123UTC'),
> ('2019-10-06 10:11:12.1234'),
> ('2019-10-06 10:11:12.12345CST'),
> ('2019-10-06 10:11:12.123456PST') t(v)
> 2019-10-06 03:11:12.123
> 2019-10-06 08:11:12.12345
> 2019-10-06 10:11:12
> 2019-10-06 10:11:12
> 2019-10-06 10:11:12.1
> 2019-10-06 10:11:12.12
> 2019-10-06 10:11:12.1234
> 2019-10-06 10:11:12.123456
> select to_timestamp('2019-10-06 10:11:12.1234567PST', 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]')
> NULL
> {code}
> Since 3.0, we use java 8 time API to parse and format timestamp values. when we create the DateTimeFormatter, we use appendPattern to create the build first, where the 'S..S' part will be parsed to a fixed-length(= 'S..S'.length). This fits the formatting part but too strict for the parsing part because the trailing zeros are very likely to be truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org