[jira] [Created] (SPARK-60161) Add tests for nested timestamp conversion in pandas type converters

["Jason Chen (Jira)" <ji...@apache.org>]

Jason Chen created SPARK-60161:
----------------------------------

             Summary: Add tests for nested timestamp conversion in pandas type converters
                 Key: SPARK-60161
                 URL: https://issues.apache.org/jira/browse/SPARK-60161
             Project: Spark
          Issue Type: Test
          Components: PySpark, Tests
    Affects Versions: 4.4.0
            Reporter: Jason Chen


The pandas type converters in {{pyspark/sql/pandas/types.py}} ({{_create_converter_to_pandas}} and {{_create_converter_from_pandas}}) handle top-level timestamp columns with a vectorized path, but timestamps nested inside complex types (e.g. {{ArrayType(TimestampType())}}) go through per-value converters that currently lack unit test coverage.

According to Codecov, several of these branches are not exercised by any test, including:
* from_pandas: timezone-aware datetime values nested in {{TimestampType}}
* from_pandas: {{TimestampLTZNanosType}} values (the whole converter)
* to_pandas: nested {{TimestampType}} with {{timestamp_utc_localized=False}}
* to_pandas: nested {{TimestampLTZNanosType}} and {{TimestampNTZType}} values

This ticket adds unit tests to {{python/pyspark/sql/tests/pandas/test_converter.py}} covering both directions for nested {{TimestampType}}, {{TimestampLTZNanosType}} and {{TimestampNTZType}}, including:
* timezone-aware vs. naive input values
* interpretation of naive values in the session timezone
* {{timestamp_utc_localized=False}}, where naive values are read in the local (system) timezone
* preservation of nanosecond precision for {{TimestampLTZNanosType}}
* None elements and None arrays

This is a test-only change with no user-facing impact.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org