You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "assaf.mendelson" <as...@rsa.com> on 2017/08/15 11:16:55 UTC
Possible bug: inconsistent timestamp behavior
Hi all,
I encountered weird behavior for timestamp. It seems that when using lit to add it to column, the timestamp goes from milliseconds representation to seconds representation:
scala> spark.range(1).withColumn("a", lit(new java.sql.Timestamp(1485503350000L)).cast("long")).show()
+---+----------+
| id| a|
+---+----------+
| 0|1485503350|
+---+----------+
scala> spark.range(1).withColumn("a", lit(1485503350000L).cast(org.apache.spark.sql.types.TimestampType).cast(org.apache.spark.sql.types.LongType)).show()
+---+-------------+
| id| a|
+---+-------------+
| 0|1485503350000|
+---+-------------+
Is this a bug or am I missing something here?
Thanks,
Assaf
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Possible-bug-inconsistent-timestamp-behavior-tp22144.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Possible bug: inconsistent timestamp behavior
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
These two are just not equivalent.
Spark SQL interprets long as seconds when casting between timestamps and
numerics, therefore
lit(1485503350000L).cast(org.apache.spark.sql.types.TimestampType)
represents 49043-09-23 21:26:400.0. This behavior is intended - see for
example https://issues.apache.org/jira/browse/SPARK-11724
java.sql.Timestamp expects milliseconds as an argument therefore lit(new
java.sql.Timestamp(1485503350000L)) represents 2017-01-27 08:49:10
.
On 15 August 2017 at 13:16, assaf.mendelson <as...@rsa.com> wrote:
> Hi all,
>
> I encountered weird behavior for timestamp. It seems that when using lit
> to add it to column, the timestamp goes from milliseconds representation to
> seconds representation:
>
>
>
>
>
> scala> spark.range(1).withColumn("a", lit(new java.sql.Timestamp(
> 1485503350000L)).cast("long")).show()
>
> +---+----------+
>
> | id| a|
>
> +---+----------+
>
> | 0|1485503350|
>
> +---+----------+
>
>
>
>
>
> scala> spark.range(1).withColumn("a", lit(1485503350000L).cast(org.
> apache.spark.sql.types.TimestampType).cast(org.apache.spark.sql.types.
> LongType)).show()
>
> +---+-------------+
>
> | id| a|
>
> +---+-------------+
>
> | 0|1485503350000|
>
> +---+-------------+
>
>
>
>
>
> Is this a bug or am I missing something here?
>
>
>
> Thanks,
>
> Assaf
>
>
>
> ------------------------------
> View this message in context: Possible bug: inconsistent timestamp
> behavior
> <http://apache-spark-developers-list.1001551.n3.nabble.com/Possible-bug-inconsistent-timestamp-behavior-tp22144.html>
> Sent from the Apache Spark Developers List mailing list archive
> <http://apache-spark-developers-list.1001551.n3.nabble.com/> at
> Nabble.com.
>
--
Z poważaniem,
Maciej Szymkiewicz