You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2025/11/13 07:18:18 UTC
(spark) branch master updated: [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new aad6c51d27ad [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
aad6c51d27ad is described below
commit aad6c51d27adb040165ea158fa4ac9fefbe9a754
Author: Yicong-Huang <17...@users.noreply.github.com>
AuthorDate: Thu Nov 13 15:18:06 2025 +0800
[SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
### What changes were proposed in this pull request?
Following up with #52680, this PR optimizes the non-Arrow path of `toPandas()` to eliminate intermediate DataFrame creation.
**Key optimizations:**
1. **Avoid intermediate DataFrame copy**
- `pd.DataFrame.from_records(rows)` → Direct column extraction via `zip(*rows)`
- 2 DataFrame creations → 1 DataFrame creation
2. **Optimize column-by-column conversion** (especially for wide tables)
- Tuples → Lists for faster Series construction
- Implicit dtype inference → Explicit `dtype=object`
- `pd.concat(axis="columns")` + column rename → `pd.concat(axis=1, keys=columns)`
- Result: **43-67% speedup for 50-100 columns**
### Why are the changes needed?
**Problem:** Current flow creates DataFrame twice:
- `rows` → `pd.DataFrame.from_records()` → temporary DataFrame → `pd.concat()` → final DataFrame
The intermediate DataFrame is immediately discarded, wasting memory. This is especially inefficient for wide tables where column-by-column overhead is significant.
### Does this PR introduce _any_ user-facing change?
No. This is a pure performance optimization with no API or behavior changes.
### How was this patch tested?
- Existing unit tests.
- Benchmark
**Benchmark setup:**
- **Hardware**: Driver memory 4GB, Executor memory 4GB
- **Configuration**: `spark.sql.execution.arrow.pyspark.enabled=false` (testing non-Arrow path)
- **Iterations**: 10 iterations per test case for statistical reliability
- **Test cases**:
- Simple (numeric columns)
- Mixed (int, string, double, boolean)
- Timestamp (date and timestamp types)
- Nested (struct and array types)
- Wide (5, 10, 50, 100 column counts)
### Performance Results
**General Benchmark** (10 iterations):
| Test Case | Rows | OLD → NEW | Speedup |
|------------|------|-----------|---------|
| simple | 1M | 1.376s → 1.383s | ≈ Tied |
| mixed | 1M | 2.396s → 2.553s | 6% slower |
| timestamp | 500K | 4.323s → 4.392s | ≈ Tied |
| nested | 100K | 0.558s → 0.580s | 4% slower |
| wide (50) | 100K | 1.458s → **1.141s** | **28% faster** 🚀 |
**Column Width Benchmark** (100K rows, 10 iterations):
| Columns | OLD → NEW | Speedup |
|---------|-----------|---------|
| 5 | 0.188s → 0.179s | 5% faster |
| 10 | 0.262s → 0.270s | ≈ Tied |
| 50 | 1.430s → **0.998s** | **43% faster** 🚀 |
| 100 | 3.320s → **1.988s** | **67% faster** 🚀 |
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-Generated-by Cursor
Closes #52897 from Yicong-Huang/SPARK-54182/refactor/avoid-intermedia-df-in-non-arrow-toPandas.
Authored-by: Yicong-Huang <17...@users.noreply.github.com>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/sql/pandas/conversion.py | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/python/pyspark/sql/pandas/conversion.py b/python/pyspark/sql/pandas/conversion.py
index a4ccf4da6e8a..e90f1c305002 100644
--- a/python/pyspark/sql/pandas/conversion.py
+++ b/python/pyspark/sql/pandas/conversion.py
@@ -18,6 +18,7 @@ import sys
from typing import (
Any,
Callable,
+ Iterator,
List,
Optional,
Union,
@@ -208,18 +209,20 @@ class PandasConversionMixin:
# Below is toPandas without Arrow optimization.
rows = self.collect()
- if len(rows) > 0:
- pdf = pd.DataFrame.from_records(
- rows, index=range(len(rows)), columns=self.columns # type: ignore[arg-type]
- )
- else:
- pdf = pd.DataFrame(columns=self.columns)
- if len(pdf.columns) > 0:
+ if len(self.columns) > 0:
timezone = sessionLocalTimeZone
struct_in_pandas = pandasStructHandlingMode
- return pd.concat(
+ # Extract columns from rows and apply converters
+ if len(rows) > 0:
+ # Use iterator to avoid materializing intermediate data structure
+ columns_data: Iterator[Any] = iter(zip(*rows))
+ else:
+ columns_data = iter([] for _ in self.schema.fields)
+
+ # Build DataFrame from columns
+ pdf = pd.concat(
[
_create_converter_to_pandas(
field.dataType,
@@ -230,13 +233,15 @@ class PandasConversionMixin:
),
error_on_duplicated_field_names=False,
timestamp_utc_localized=False,
- )(pser)
- for (_, pser), field in zip(pdf.items(), self.schema.fields)
+ )(pd.Series(col_data, dtype=object))
+ for col_data, field in zip(columns_data, self.schema.fields)
],
- axis="columns",
+ axis=1,
+ keys=self.columns,
)
- else:
return pdf
+ else:
+ return pd.DataFrame(columns=[], index=range(len(rows)))
def toArrow(self) -> "pa.Table":
from pyspark.sql.dataframe import DataFrame
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org