You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Spenserrrr (via GitHub)" <gi...@apache.org> on 2026/09/28 20:35:56 UTC
Re: [PR] [SPARK-59624][PYTHON][FOLLOWUP] Support an optional schema in concat_batches for empty input [spark]
Spenserrrr commented on code in PR #59107:
URL: https://github.com/apache/spark/pull/59107#discussion_r4126775784
##########
python/pyspark/sql/conversion.py:
##########
@@ -185,14 +185,17 @@ def select_columns(cls, batch: "pa.RecordBatch", column_indices: list[int]) -> "
def concat_batches(cls, batches: Iterable["pa.RecordBatch"]) -> "pa.RecordBatch":
"""Concatenate same-schema RecordBatches by row.
- A single batch is returned unchanged. PyArrow before 19.0.0 has no ``concat_batches``;
- the fallback concatenates the equivalent StructArrays and converts the result back to a
+ ``batches`` is materialized into a tuple, so a caller can pass an iterator without
Review Comment:
Updated, the docstring now says any iterable is accepted and consumed once.
##########
python/pyspark/sql/conversion.py:
##########
@@ -185,14 +185,17 @@ def select_columns(cls, batch: "pa.RecordBatch", column_indices: list[int]) -> "
def concat_batches(cls, batches: Iterable["pa.RecordBatch"]) -> "pa.RecordBatch":
"""Concatenate same-schema RecordBatches by row.
- A single batch is returned unchanged. PyArrow before 19.0.0 has no ``concat_batches``;
- the fallback concatenates the equivalent StructArrays and converts the result back to a
+ ``batches`` is materialized into a tuple, so a caller can pass an iterator without
+ building a list first. An empty input returns an empty RecordBatch with no columns, and a
+ single batch is returned unchanged. PyArrow before 19.0.0 has no ``concat_batches``; the
+ fallback concatenates the equivalent StructArrays and converts the result back to a
RecordBatch.
"""
import pyarrow as pa
batches = tuple(batches)
- assert batches
+ if not batches:
+ return pa.RecordBatch.from_arrays([], [])
Review Comment:
Done. An empty input now raises `PySparkValueError` unless a `schema` is passed, in which case it returns a zero-row batch with that schema, the same contract as `pa.Table.from_batches`. A given schema must match every batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org