You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/22 22:34:00 UTC
[jira] [Updated] (SPARK-60484) CAST(string AS INTERVAL SECOND) drops the minus sign of values between -1 and 0 seconds
[ https://issues.apache.org/jira/browse/SPARK-60484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-60484:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> CAST(string AS INTERVAL SECOND) drops the minus sign of values between -1 and 0 seconds
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-60484
> URL: https://issues.apache.org/jira/browse/SPARK-60484
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:sql}
> SELECT CAST('-0.5' AS INTERVAL SECOND);
> -- expected: INTERVAL '-00.5' SECOND
> -- actual: INTERVAL '00.5' SECOND
> SELECT CAST('-1.5' AS INTERVAL SECOND);
> -- INTERVAL '-01.5' SECOND (correct)
> {code}
> Any value between -1 and 0 seconds loses its sign. Since the JSON and CSV readers parse INTERVAL SECOND with the same code, writing such a value to JSON or CSV and reading it back also turns -0.5 s into +0.5 s:
> {code:python}
> df = spark.sql("SELECT INTERVAL '-0.5' SECOND AS x")
> df.write.mode("overwrite").json("/tmp/iv")
> spark.read.schema(df.schema).json("/tmp/iv").first().x
> # datetime.timedelta(microseconds=500000), expected -500000
> {code}
> Cause: in {{IntervalUtils.parseSecondNano}}, the sign of the fraction is taken from the parsed whole seconds ({{parseNanos(nanosStr, seconds < 0)}}). For "-00.5" the whole seconds are 0, so the fraction is added as positive. The interval literal {{INTERVAL '-0.5' SECOND}} goes through the parser and is correct.
> Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org