[jira] [Created] (SPARK-55312) Always infer Spark types in createDataFrame when schema is not provided

["Yicong Huang (Jira)" <ji...@apache.org>]

Yicong Huang created SPARK-55312:
------------------------------------

             Summary: Always infer Spark types in createDataFrame when schema is not provided
                 Key: SPARK-55312
                 URL: https://issues.apache.org/jira/browse/SPARK-55312
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Yicong Huang



Currently, when {{createDataFrame}} is called without a schema, the {{spark_types}} list contains {{None}} for non-timestamp columns:

{code:python}
# pyspark/sql/pandas/conversion.py
spark_types = [
    TimestampType()
    if is_datetime64_dtype(t) or isinstance(t, pd.DatetimeTZDtype)
    else None  # Non-timestamp columns get None
    for t in pdf.dtypes
]
{code}

This {{None}} is passed to the serializer, which then lets PyArrow infer the type. This creates inconsistency because the Spark type is not explicitly defined.

The fix is to always infer Spark types from Arrow types when no schema is provided:

{code:python}
arrow_schema = pa.Schema.from_pandas(pdf, preserve_index=False)
spark_types = [
    TimestampType()
    if is_datetime64_dtype(t) or isinstance(t, pd.DatetimeTZDtype)
    else from_arrow_type(field.type, prefer_timestamp_ntz)
    for t, field in zip(pdf.dtypes, arrow_schema)
]
{code}

This ensures {{spark_type}} is always defined, which:
1. Makes the code more consistent
2. Enables downstream code to always have a valid Spark type
3. Prepares for making {{spark_type}} a required parameter in serializers

Files to change:
- {{pyspark/sql/pandas/conversion.py}}
- {{pyspark/sql/connect/session.py}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org