[jira] [Updated] (SPARK-35826) Timestamp object inserts extra conversion to local time when extracting row

["Jennifer Webster (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-35826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jennifer Webster updated SPARK-35826:
-------------------------------------
    Attachment: timezone_MWE.py

> Timestamp object inserts extra conversion to local time when extracting row
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-35826
>                 URL: https://issues.apache.org/jira/browse/SPARK-35826
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 3.0.1, 3.0.2
>         Environment: conda installation of pyspark 3.0.1 or 3.0.2 with python 3.8 
>            Reporter: Jennifer Webster
>            Priority: Major
>         Attachments: timezone_MWE.py
>
>
> When converting our code from spark 2.4 to spark 3.0, we noticed that several of our time related tests were failing.  Most of these were fixed by setting the spark session and java driver/executor times to be identical and UTC.  However, the timezone conversion tests were still failing because when we used df.first() or .collect() to extract a single row of the dataframe, the local computer time was being introduced to the datetime object and inserting an extra conversion. 
> Below is my MWE. 
> These tests passed when we were working wtih pyspark 2.4.6/7, but have failed in pyspark 3.0.1/2.  We can not currently upgrade to python 3.9 because we need pyarrow support and we can not move to pyspark 3.1 because apache sedona does not support that.
> {code:java}
> // time_zone_MWE.py
> import datetime
> import pyspark.sql.functions as sf
> from pyspark.conf import SparkConf
> from pyspark.sql import SparkSession
> if __name__ == "__main__":    
>    conf = SparkConf()    
>    arguments = {
>         "spark.master": "local[2]",
>         "spark.sql.parquet.enableVectorizedReader": "False",
>         "spark.sql.execution.arrow.pyspark.enabled": "True",
>         "spark.sql.execution.arrow.pyspark.fallback.enabled": "True",
>         "spark.submit.deployMode": "client",
>         "spark.sql.session.timeZone": "UTC",
>         "spark.sql.legacy.timeParserPolicy": "LEGACY",
>         "spark.cleaner.referenceTracking.cleanCheckpoints": "True",
>         "spark.driver.extraJavaOptions": "-Duser.timezone=UTC",
>         "spark.executor.extraJavaOptions": "-Duser.timezone=UTC",
>     }
>     conf.setAll(arguments.items())
>     spark = SparkSession.builder.config(conf=conf).getOrCreate()
>     # Test Dataframe with local time and the timezone
>     timezone_data = spark.createDataFrame(
>         [("2008-02-02 13:36:44", "Asia/Vladivostok")], ["datetime", "timezone"]
>     )
>     # expected conversion should be UTC+10 should be 2008-02-02 03:36:44
>     df = timezone_data.withColumn(
>         "utc_ts",
>         sf.to_utc_timestamp(
>             timezone_data.datetime.cast("timestamp"), timezone_data.timezone
>         ),
>     )
>     df = df.withColumn("utc_int", df.utc_ts.cast("int"))
>     # This shows correct time
>     print(df.show())
>     # This shows a local computer system offset
>     print(df.first())
>     # Comparing the two integers will work
>     assert (
>         int(
>             datetime.datetime(
>                 2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
>             ).timestamp()
>         )
>         == df.first()["utc_int"]
>     )
>     # Comparing the two datetime objects will not (but worked in pyspark 2.4.6)
>     assert df.first()["utc_ts"] == datetime.datetime(
>         2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
>     )
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org