You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@spark.apache.org by ru...@apache.org on 2025/09/24 11:31:23 UTC

(spark) branch master updated (b005f5692842 -> 4418d6ec567e)

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


    from b005f5692842 [SPARK-53689][BUILD][FOLLOW-UP] Check if RELEASE_VERSION is already set properly
     add 4418d6ec567e [SPARK-53562][PYTHON] Limit Arrow batch sizes in `applyInArrow` and `applyInPandas`

No new revisions were added by this update.

Summary of changes:
 python/pyspark/sql/pandas/serializers.py           |  87 ++++++++++++++
 .../sql/tests/arrow/test_arrow_grouped_map.py      |  40 ++++++-
 .../connect/arrow/test_parity_arrow_grouped_map.py |   4 +-
 .../pandas/test_parity_pandas_grouped_map.py       |   4 +-
 .../sql/tests/pandas/test_pandas_grouped_map.py    |  50 ++++++--
 python/pyspark/worker.py                           |  13 +-
 .../sql/execution/arrow/ArrowWriterWrapper.scala   |  84 +++++++++++++
 .../python/FlatMapGroupsInBatchExec.scala          |   2 +-
 .../sql/execution/python/PythonArrowInput.scala    | 132 ++++++++++++++++++---
 9 files changed, 379 insertions(+), 37 deletions(-)
 create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org