You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2020/03/16 22:42:00 UTC
[jira] [Commented] (SPARK-31150) Parsing seconds fraction with
variable length for timestamp
[ https://issues.apache.org/jira/browse/SPARK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060513#comment-17060513 ]
Dongjoon Hyun commented on SPARK-31150:
---------------------------------------
Hi, [~Qin Yao]. Is this a bug in 3.0.0 or an improvement in 3.1.0?
> Parsing seconds fraction with variable length for timestamp
> -----------------------------------------------------------
>
> Key: SPARK-31150
> URL: https://issues.apache.org/jira/browse/SPARK-31150
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Kent Yao
> Priority: Major
>
> This JIRA is to support parsing timestamp values with variable length second fraction parts.
> e.g. 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]' can parse timestamp with 0~6 digit-length second fraction but fail >=7
> {code:java}
> select to_timestamp(v, 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]') from values
> ('2019-10-06 10:11:12.'),
> ('2019-10-06 10:11:12.0'),
> ('2019-10-06 10:11:12.1'),
> ('2019-10-06 10:11:12.12'),
> ('2019-10-06 10:11:12.123UTC'),
> ('2019-10-06 10:11:12.1234'),
> ('2019-10-06 10:11:12.12345CST'),
> ('2019-10-06 10:11:12.123456PST') t(v)
> 2019-10-06 03:11:12.123
> 2019-10-06 08:11:12.12345
> 2019-10-06 10:11:12
> 2019-10-06 10:11:12
> 2019-10-06 10:11:12.1
> 2019-10-06 10:11:12.12
> 2019-10-06 10:11:12.1234
> 2019-10-06 10:11:12.123456
> select to_timestamp('2019-10-06 10:11:12.1234567PST', 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]')
> NULL
> {code}
> Since 3.0, we use java 8 time API to parse and format timestamp values. when we create the DateTimeFormatter, we use appendPattern to create the build first, where the 'S..S' part will be parsed to a fixed-length(= 'S..S'.length). This fits the formatting part but too strict for the parsing part because the trailing zeros are very likely to be truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org