You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "tonghuaroot (via GitHub)" <gi...@apache.org> on 2026/06/07 01:38:15 UTC
[PR] [SPARK-57294][PS] Support DataFrame.combine in fallback mode [spark]
tonghuaroot opened a new pull request, #56359:
URL: https://github.com/apache/spark/pull/56359
### What changes were proposed in this pull request?
This PR enables `DataFrame.combine` for pandas-on-Spark through the
`compute.pandas_fallback` path. Previously `combine` was declared via
`_unsupported_function`, so it always raised `PandasNotImplementedError`.
This PR adds a `_combine_fallback` method to
`pyspark.pandas.frame.DataFrame`, mirroring the existing
`_asof_fallback` / `_set_axis_fallback` sibling methods, so that
`__getattr__` dispatches `combine` through the generic
`_build_fallback_method` when the fallback option is enabled.
It also adds tests covering both the disabled (raises
`PandasNotImplementedError`) and the fallback-enabled behavior, plus the
Spark Connect parity test, and registers them in
`dev/sparktestsupport/modules.py`.
JIRA: https://issues.apache.org/jira/browse/SPARK-57294
### Why are the changes needed?
`combine` is a useful pandas DataFrame API that was unsupported on
pandas-on-Spark even when users opted into `compute.pandas_fallback`.
It is a sound fallback candidate for the same reasons as the existing
asof / set_axis fallbacks: its result is an ordinary single-level-index
DataFrame whose column dtypes (for example int64) map cleanly onto Spark
types, so the generic fallback round-trip through
`ps.from_pandas` / `as_spark_type` succeeds. Wiring it through fallback
closes a gap in the pandas-on-Spark fallback coverage and gives users an
explicit, opt-in way to run `combine`.
### Does this PR introduce _any_ user-facing change?
Yes. With `compute.pandas_fallback` enabled, calling
`DataFrame.combine` on a pandas-on-Spark DataFrame now executes via the
pandas fallback path and returns a result instead of raising
`PandasNotImplementedError`. A `PandasAPIOnSparkAdviceWarning` is emitted
to indicate the call ran in fallback mode. When the option is disabled
(the default), the behavior is unchanged and `PandasNotImplementedError`
is still raised.
### How was this patch tested?
Added `pyspark.pandas.tests.frame.test_combine` and the Spark Connect
parity test `pyspark.pandas.tests.connect.frame.test_parity_combine`,
both registered in `dev/sparktestsupport/modules.py`. The classic test
covers two cases:
- `test_disabled`: without `compute.pandas_fallback`, `combine` raises
`PandasNotImplementedError`.
- `test_fallback`: with the option enabled, `combine` (including the
`overwrite=False` case) produces results equal to pandas, asserted
with `assert_eq` (values and dtypes).
Ran `test_combine` against a real local SparkSession:
```
$ python -m pytest python/pyspark/pandas/tests/frame/test_combine.py -v
collected 4 items
... test_assert_classic_mode PASSED
... CombineTests::test_assert_classic_mode PASSED
... CombineTests::test_disabled PASSED
... CombineTests::test_fallback PASSED
4 passed in 11.32s
```
Environment: PySpark master (based on commit c082f824), pandas 2.2.3,
PyArrow as bundled, OpenJDK 17.0.18, Python 3.11. The
`PandasAPIOnSparkAdviceWarning: combine is executed in fallback mode`
message confirms the call exercised the fallback path.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (Anthropic) Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57294][PS] Support DataFrame.combine in fallback mode [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #56359: [SPARK-57294][PS] Support DataFrame.combine in fallback mode
URL: https://github.com/apache/spark/pull/56359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57294][PS] Support DataFrame.combine in fallback mode [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56359:
URL: https://github.com/apache/spark/pull/56359#issuecomment-4646014865
cc @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57294][PS] Support DataFrame.combine in fallback mode [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #56359:
URL: https://github.com/apache/spark/pull/56359#issuecomment-4648204041
thanks, merged into master/4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org