You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/03/14 10:10:00 UTC
[jira] [Assigned] (SPARK-38520) Overflow occurs when reading ANSI day time interval from CSV file
[ https://issues.apache.org/jira/browse/SPARK-38520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-38520:
------------------------------------
Assignee: Apache Spark
> Overflow occurs when reading ANSI day time interval from CSV file
> -----------------------------------------------------------------
>
> Key: SPARK-38520
> URL: https://issues.apache.org/jira/browse/SPARK-38520
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: chong
> Assignee: Apache Spark
> Priority: Major
>
> *Problem:*
> Overflow occurs when reading the following positive intervals, the results become to negative
> interval '106751992' day => INTERVAL '-106751990' DAY
> INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR
> interval '153722867281' minute => INTERVAL '-153722867280' MINUTE
>
> *Reroduce:*
> {code}
> // days overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
> scala> spark.read.csv(path).show(false)
> +------------------------+
> |_c0 |
> +------------------------+
> |interval '106751992' day|
> +------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +-------------------------+
> |c1 |
> +-------------------------+
> |INTERVAL '-106751990' DAY|
> +-------------------------+
> // hour overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
> scala> spark.read.csv(path).show(false)
> +----------------------------+
> |_c0 |
> +----------------------------+
> |INTERVAL +'+2562047789' hour|
> +----------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +---------------------------+
> |c1 |
> +---------------------------+
> |INTERVAL '-2562047787' HOUR|
> +---------------------------+
> // minute overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.MINUTE, DayTimeIntervalType.MINUTE))))
> scala> spark.read.csv(path).show(false)
> +------------------------------+
> |_c0 |
> +------------------------------+
> |interval '153722867281' minute|
> +------------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +-------------------------------+
> |c1 |
> +-------------------------------+
> |INTERVAL '-153722867280' MINUTE|
> +-------------------------------+
> {code}
> *others:*
> Also check the negative value is read to positive.
>
> others:
> should check the negative also,
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org