[jira] [Created] (SPARK-35826) Timestamp object inserts extra conversion to local time when extracting row

["Jennifer Webster (Jira)" <ji...@apache.org>]

Jennifer Webster created SPARK-35826:
----------------------------------------

             Summary: Timestamp object inserts extra conversion to local time when extracting row
                 Key: SPARK-35826
                 URL: https://issues.apache.org/jira/browse/SPARK-35826
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.0.2, 3.0.1
         Environment: conda installation of pyspark 3.0.1 or 3.0.2 with python 3.8 
            Reporter: Jennifer Webster


When converting our code from spark 2.4 to spark 3.0, we noticed that several of our time related tests were failing.  Most of these were fixed by setting the spark session and java driver/executor times to be identical and UTC.  However, the timezone conversion tests were still failing because when we used df.first() or .collect() to extract a single row of the dataframe, the local computer time was being introduced to the datetime object and inserting an extra conversion. 

Below is my MWE. 

These tests passed when we were working wtih pyspark 2.4.6/7, but have failed in pyspark 3.0.1/2.  We can not currently upgrade to python 3.9 because we need pyarrow support and we can not move to pyspark 3.1 because apache sedona does not support that.
{code:java}
// time_zone_MWE.py

import datetime
import pyspark.sql.functions as sf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

if __name__ == "__main__":    
   conf = SparkConf()    
   arguments = {
        "spark.master": "local[2]",
        "spark.sql.parquet.enableVectorizedReader": "False",
        "spark.sql.execution.arrow.pyspark.enabled": "True",
        "spark.sql.execution.arrow.pyspark.fallback.enabled": "True",
        "spark.submit.deployMode": "client",
        "spark.sql.session.timeZone": "UTC",
        "spark.sql.legacy.timeParserPolicy": "LEGACY",
        "spark.cleaner.referenceTracking.cleanCheckpoints": "True",
        "spark.driver.extraJavaOptions": "-Duser.timezone=UTC",
        "spark.executor.extraJavaOptions": "-Duser.timezone=UTC",
    }
    conf.setAll(arguments.items())
    spark = SparkSession.builder.config(conf=conf).getOrCreate()
    # Test Dataframe with local time and the timezone
    timezone_data = spark.createDataFrame(
        [("2008-02-02 13:36:44", "Asia/Vladivostok")], ["datetime", "timezone"]
    )
    # expected conversion should be UTC+10 should be 2008-02-02 03:36:44
    df = timezone_data.withColumn(
        "utc_ts",
        sf.to_utc_timestamp(
            timezone_data.datetime.cast("timestamp"), timezone_data.timezone
        ),
    )
    df = df.withColumn("utc_int", df.utc_ts.cast("int"))
    # This shows correct time
    print(df.show())
    # This shows a local computer system offset
    print(df.first())
    # Comparing the two integers will work
    assert (
        int(
            datetime.datetime(
                2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
            ).timestamp()
        )
        == df.first()["utc_int"]
    )
    # Comparing the two datetime objects will not (but worked in pyspark 2.4.6)
    assert df.first()["utc_ts"] == datetime.datetime(
        2008, 2, 2, 3, 36, 44, tzinfo=datetime.timezone.utc
    )

{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org