You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2026/08/05 06:48:13 UTC
(spark) branch branch-4.1 updated: [SPARK-58529][4.1][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for cross-version old-client compatibility
This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new eefdff7d30d6 [SPARK-58529][4.1][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for cross-version old-client compatibility
eefdff7d30d6 is described below
commit eefdff7d30d6a1056c2d25ea66ca9f54d63aae5c
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Tue Aug 4 23:48:02 2026 -0700
[SPARK-58529][4.1][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for cross-version old-client compatibility
### What changes were proposed in this pull request?
This backports the test-side change of SPARK-58529 to this maintenance branch. It relaxes the `test_vectorized_udf_invalid_length` assertion in `test_pandas_udf_scalar.py` to accept both this branch's `SCHEMA_MISMATCH_FOR_PANDAS_UDF` message (`Result vector from pandas_udf was not the required length`) and the `RESULT_ROWS_MISMATCH` message a newer server raises (`The number of output rows ... must match the number of input rows`).
No production code changes -- this branch continues to raise its existing message.
### Why are the changes needed?
The `pyspark-connect-old-client` cross-version job clones a maintenance branch's tests and runs them against a newer (master) server. It currently pins `branch-4.0`, but the base is expected to advance to `branch-4.1` / `branch-4.2`. When it does, an assertion pinned to the old pandas-specific substring would fail against a master server that no longer emits it (after SPARK-58529). Relaxing the assertion now -- to match both messages -- keeps this branch's own CI green today and preve [...]
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The relaxed regex was verified to match both this branch's message and the newer `RESULT_ROWS_MISMATCH` message. Existing `test_vectorized_udf_invalid_length` continues to run.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57776 from Yicong-Huang/backport-58529-branch-4.1.
Authored-by: Yicong Huang <17...@users.noreply.github.com>
Signed-off-by: Yicong Huang <17...@users.noreply.github.com>
---
python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
index 637bc6854a6a..a19194011ff8 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py
@@ -677,8 +677,13 @@ class ScalarPandasUDFTestsMixin:
def check_vectorized_udf_invalid_length(self):
df = self.spark.range(10)
raise_exception = pandas_udf(lambda _: pd.Series(1), LongType())
+ # Accept both this branch's SCHEMA_MISMATCH_FOR_PANDAS_UDF message and the
+ # RESULT_ROWS_MISMATCH message a newer server raises (SPARK-58529), so the
+ # cross-version old-client job passes against a master server.
with self.assertRaisesRegex(
- Exception, "Result vector from pandas_udf was not the required length"
+ Exception,
+ "Result vector from pandas_udf was not the required length"
+ "|The number of output rows.*must match the number of input rows",
):
df.select(raise_exception(col("id"))).collect()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org