[jira] [Created] (SPARK-55349) Consolidate pandas-to-Arrow conversion utilities in serializers

["Yicong Huang (Jira)" <ji...@apache.org>]

Yicong Huang created SPARK-55349:
------------------------------------

             Summary: Consolidate pandas-to-Arrow conversion utilities in serializers
                 Key: SPARK-55349
                 URL: https://issues.apache.org/jira/browse/SPARK-55349
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Yicong Huang



The pandas UDF serializers contain significant code duplication for converting pandas data to Arrow format. Multiple `_create_batch` and `_create_array` methods exist across different serializer classes with nearly identical logic:

{code:python}
# ArrowStreamPandasSerializer
def _create_batch(self, series):
    arrs = []
    for s, t in series:
        # ... conversion logic ...
    return pa.RecordBatch.from_arrays(arrs, ...)

# ArrowStreamPandasUDFSerializer  
def _create_batch(self, series):
    # ... similar conversion logic ...

# ArrowStreamPandasUDTFSerializer
def _create_array(self, series, spark_type):
    # ... conversion logic ...
{code}

This duplication makes the code harder to maintain and increases the risk of inconsistent behavior.

Proposal: Extract the common conversion logic into a dedicated `PandasToArrowConversion` class in `conversion.py`:

{code:python}
class PandasToArrowConversion:
    @classmethod
    def dataframe_to_batch(cls, data, schema, ...) -> pa.RecordBatch: ...
    
    @classmethod
    def series_to_array(cls, series, spark_type, ...) -> pa.Array: ...
{code}

This reduces code duplication and provides a single, well-tested conversion path.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org