You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/03/19 06:27:02 UTC
(spark) branch master updated: [SPARK-55059][PYTHON] Remove empty table workaround in toPandas
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f362c7eb44d8 [SPARK-55059][PYTHON] Remove empty table workaround in toPandas
f362c7eb44d8 is described below
commit f362c7eb44d860fbd57d2dfd6c293f8ddb28375a
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Thu Mar 19 15:26:51 2026 +0900
[SPARK-55059][PYTHON] Remove empty table workaround in toPandas
### What changes were proposed in this pull request?
Remove the SPARK-51112 workaround in `_convert_arrow_table_to_pandas()` that bypassed PyArrow's `to_pandas()` for empty tables.
### Why are the changes needed?
The workaround was added because arrow-java's `ListVector.getBufferSizeFor(0)` returned 0, causing the offset buffer to be omitted for empty nested arrays in IPC serialization, which led to a segmentation fault in PyArrow.
This has been fixed upstream in arrow-java 19.0.0 ([apache/arrow-java#343](https://github.com/apache/arrow-java/issues/343)), which Spark adopted in SPARK-56000 (PR #54820). The workaround is no longer necessary.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test `test_to_pandas_for_empty_df_with_nested_array_columns` passes.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53824 from Yicong-Huang/SPARK-55059/refactor/remove-empty-table-workaround.
Authored-by: Yicong Huang <17...@users.noreply.github.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/pandas/conversion.py | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/python/pyspark/sql/pandas/conversion.py b/python/pyspark/sql/pandas/conversion.py
index d3af60666090..3f5d68d10452 100644
--- a/python/pyspark/sql/pandas/conversion.py
+++ b/python/pyspark/sql/pandas/conversion.py
@@ -254,16 +254,8 @@ def _convert_arrow_table_to_pandas(
error_on_duplicated_field_names = True
struct_handling_mode = "dict"
- # SPARK-51112: If the table is empty, we avoid using pyarrow to_pandas to create the
- # DataFrame, as it may fail with a segmentation fault.
- if arrow_table.num_rows == 0:
- # For empty tables, create empty Series to preserve dtypes
- column_data = (
- pd.Series([], name=temp_col_names[i], dtype="object") for i in range(len(schema.fields))
- )
- else:
- # For non-empty tables, convert arrow columns directly
- column_data = (arrow_col.to_pandas(**pandas_options) for arrow_col in arrow_table.columns)
+ # Convert arrow columns to pandas Series
+ column_data = (arrow_col.to_pandas(**pandas_options) for arrow_col in arrow_table.columns)
# Apply Spark-specific type converters to each column
pdf = pd.concat(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org