You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2026/03/20 23:33:17 UTC
[PR] [SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov [spark]
ueshin opened a new pull request, #54933:
URL: https://github.com/apache/spark/pull/54933
### What changes were proposed in this pull request?
This PR updates `pyspark.pandas.Series.cov` to use pandas' `is_numeric_dtype` instead of `np.issubdtype(..., np.number)` when validating the input Series dtypes.
The previous NumPy-based check works for plain NumPy dtypes, but it raises a `TypeError` for pandas extension dtypes such as `StringDtype`, which causes `Series.cov` to fail before it reaches the intended `unsupported dtype` error path.
This PR also updates the related `Series.cov` tests to:
- accept the dtype name shown by both pandas 2 (`object`) and pandas 3 (`str`) for non-numeric string inputs
- add extension dtype coverage for nullable `Float64`
- add a string extension dtype case using pandas `"string"`
### Why are the changes needed?
With pandas 3, `Series(["a", "b", "c"]).dtype` is a pandas extension dtype (`StringDtype`), and `np.issubdtype(self.dtype, np.number)` raises:
`TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type`
That means `Series.cov` fails in dtype validation instead of returning the expected `TypeError("unsupported dtype: ...")`.
Using `is_numeric_dtype` is more appropriate here because it is pandas-aware and works with both NumPy dtypes and pandas extension dtypes. This keeps the existing behavior for numeric inputs while making the validation path robust across pandas versions.
### Does this PR introduce _any_ user-facing change?
Yes.
For pandas extension dtypes such as pandas 3 string dtype, `pyspark.pandas.Series.cov` now raises the intended `TypeError("unsupported dtype: ...")` instead of failing earlier with a NumPy dtype interpretation error.
### How was this patch tested?
Added the related tests and the other existing tests should pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex GPT-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #54933: [SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov
URL: https://github.com/apache/spark/pull/54933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #54933:
URL: https://github.com/apache/spark/pull/54933#issuecomment-4101452817
cc @gaogaotiantian @HyukjinKwon @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #54933:
URL: https://github.com/apache/spark/pull/54933#issuecomment-4102768059
Merged to master. Thank you, @ueshin .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org