You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2025/11/13 07:18:18 UTC

(spark) branch master updated: [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new aad6c51d27ad [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
aad6c51d27ad is described below

commit aad6c51d27adb040165ea158fa4ac9fefbe9a754
Author: Yicong-Huang <17...@users.noreply.github.com>
AuthorDate: Thu Nov 13 15:18:06 2025 +0800

    [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
    
    ### What changes were proposed in this pull request?
    
    Following up with #52680, this PR optimizes the non-Arrow path of `toPandas()` to eliminate intermediate DataFrame creation.
    
    **Key optimizations:**
    
    1. **Avoid intermediate DataFrame copy**
       - `pd.DataFrame.from_records(rows)` → Direct column extraction via `zip(*rows)`
       - 2 DataFrame creations → 1 DataFrame creation
    
    2. **Optimize column-by-column conversion** (especially for wide tables)
       - Tuples → Lists for faster Series construction
       - Implicit dtype inference → Explicit `dtype=object`
       - `pd.concat(axis="columns")` + column rename → `pd.concat(axis=1, keys=columns)`
       - Result: **43-67% speedup for 50-100 columns**
    
    ### Why are the changes needed?
    
    **Problem:** Current flow creates DataFrame twice:
    - `rows` → `pd.DataFrame.from_records()` → temporary DataFrame → `pd.concat()` → final DataFrame
    
    The intermediate DataFrame is immediately discarded, wasting memory. This is especially inefficient for wide tables where column-by-column overhead is significant.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a pure performance optimization with no API or behavior changes.
    
    ### How was this patch tested?
    
    - Existing unit tests.
    - Benchmark
    
    **Benchmark setup:**
    - **Hardware**: Driver memory 4GB, Executor memory 4GB
    - **Configuration**: `spark.sql.execution.arrow.pyspark.enabled=false` (testing non-Arrow path)
    - **Iterations**: 10 iterations per test case for statistical reliability
    - **Test cases**:
      - Simple (numeric columns)
      - Mixed (int, string, double, boolean)
      - Timestamp (date and timestamp types)
      - Nested (struct and array types)
      - Wide (5, 10, 50, 100 column counts)
    
    ### Performance Results
    
    **General Benchmark** (10 iterations):
    
    | Test Case  | Rows | OLD → NEW | Speedup |
    |------------|------|-----------|---------|
    | simple     | 1M   | 1.376s → 1.383s | ≈ Tied |
    | mixed      | 1M   | 2.396s → 2.553s | 6% slower |
    | timestamp  | 500K | 4.323s → 4.392s | ≈ Tied |
    | nested     | 100K | 0.558s → 0.580s | 4% slower |
    | wide (50)  | 100K | 1.458s → **1.141s** | **28% faster** 🚀 |
    
    **Column Width Benchmark** (100K rows, 10 iterations):
    
    | Columns | OLD → NEW | Speedup |
    |---------|-----------|---------|
    | 5       | 0.188s → 0.179s | 5% faster |
    | 10      | 0.262s → 0.270s | ≈ Tied |
    | 50      | 1.430s → **0.998s** | **43% faster** 🚀 |
    | 100     | 3.320s → **1.988s** | **67% faster** 🚀 |
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes. Co-Generated-by Cursor
    
    Closes #52897 from Yicong-Huang/SPARK-54182/refactor/avoid-intermedia-df-in-non-arrow-toPandas.
    
    Authored-by: Yicong-Huang <17...@users.noreply.github.com>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 python/pyspark/sql/pandas/conversion.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/python/pyspark/sql/pandas/conversion.py b/python/pyspark/sql/pandas/conversion.py
index a4ccf4da6e8a..e90f1c305002 100644
--- a/python/pyspark/sql/pandas/conversion.py
+++ b/python/pyspark/sql/pandas/conversion.py
@@ -18,6 +18,7 @@ import sys
 from typing import (
     Any,
     Callable,
+    Iterator,
     List,
     Optional,
     Union,
@@ -208,18 +209,20 @@ class PandasConversionMixin:
 
         # Below is toPandas without Arrow optimization.
         rows = self.collect()
-        if len(rows) > 0:
-            pdf = pd.DataFrame.from_records(
-                rows, index=range(len(rows)), columns=self.columns  # type: ignore[arg-type]
-            )
-        else:
-            pdf = pd.DataFrame(columns=self.columns)
 
-        if len(pdf.columns) > 0:
+        if len(self.columns) > 0:
             timezone = sessionLocalTimeZone
             struct_in_pandas = pandasStructHandlingMode
 
-            return pd.concat(
+            # Extract columns from rows and apply converters
+            if len(rows) > 0:
+                # Use iterator to avoid materializing intermediate data structure
+                columns_data: Iterator[Any] = iter(zip(*rows))
+            else:
+                columns_data = iter([] for _ in self.schema.fields)
+
+            # Build DataFrame from columns
+            pdf = pd.concat(
                 [
                     _create_converter_to_pandas(
                         field.dataType,
@@ -230,13 +233,15 @@ class PandasConversionMixin:
                         ),
                         error_on_duplicated_field_names=False,
                         timestamp_utc_localized=False,
-                    )(pser)
-                    for (_, pser), field in zip(pdf.items(), self.schema.fields)
+                    )(pd.Series(col_data, dtype=object))
+                    for col_data, field in zip(columns_data, self.schema.fields)
                 ],
-                axis="columns",
+                axis=1,
+                keys=self.columns,
             )
-        else:
             return pdf
+        else:
+            return pd.DataFrame(columns=[], index=range(len(rows)))
 
     def toArrow(self) -> "pa.Table":
         from pyspark.sql.dataframe import DataFrame


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org