You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ze...@apache.org on 2021/09/02 13:11:26 UTC
[spark] branch branch-3.2 updated: [SPARK-36617][PYTHON] Fix type
hints for `approxQuantile` to support multi-column version
This is an automated email from the ASF dual-hosted git repository.
zero323 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 11d10fc [SPARK-36617][PYTHON] Fix type hints for `approxQuantile` to support multi-column version
11d10fc is described below
commit 11d10fc994bb7e801e527ea765e1674a5a35d446
Author: Cary Lee <ca...@amperity.com>
AuthorDate: Thu Sep 2 15:02:40 2021 +0200
[SPARK-36617][PYTHON] Fix type hints for `approxQuantile` to support multi-column version
### What changes were proposed in this pull request?
Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied.
### Why are the changes needed?
The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see [the approxQuantile docs](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.approxQuantile.html).) This change was also introduced to pyspark-stubs (https://github.com/zero323/pyspark-stubs/pull/552). zero323 asked me to open a PR for the upstream change.
### Does this PR introduce _any_ user-facing change?
This change only affects type hints - it brings the `approxQuantile` type hints up to date with the actual code.
### How was this patch tested?
Ran `./dev/lint-python`.
Closes #33880 from carylee/master.
Authored-by: Cary Lee <ca...@amperity.com>
Signed-off-by: zero323 <ms...@gmail.com>
(cherry picked from commit 37f5ab07fa2343e77ae16b6460898ecbee4b3faf)
Signed-off-by: zero323 <ms...@gmail.com>
---
python/pyspark/sql/dataframe.pyi | 27 +++++++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/sql/dataframe.pyi b/python/pyspark/sql/dataframe.pyi
index 9e762bf..d43c311 100644
--- a/python/pyspark/sql/dataframe.pyi
+++ b/python/pyspark/sql/dataframe.pyi
@@ -238,12 +238,20 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
value: OptionalPrimitiveType,
subset: Optional[List[str]] = ...,
) -> DataFrame: ...
+ @overload
def approxQuantile(
self,
- col: Union[str, Tuple[str, ...], List[str]],
- probabilities: Union[List[float], Tuple[float, ...]],
- relativeError: float
+ col: str,
+ probabilities: Union[List[float], Tuple[float]],
+ relativeError: float,
) -> List[float]: ...
+ @overload
+ def approxQuantile(
+ self,
+ col: Union[List[str], Tuple[str]],
+ probabilities: Union[List[float], Tuple[float]],
+ relativeError: float,
+ ) -> List[List[float]]: ...
def corr(self, col1: str, col2: str, method: Optional[str] = ...) -> float: ...
def cov(self, col1: str, col2: str) -> float: ...
def crosstab(self, col1: str, col2: str) -> DataFrame: ...
@@ -316,9 +324,20 @@ class DataFrameNaFunctions:
class DataFrameStatFunctions:
df: DataFrame
def __init__(self, df: DataFrame) -> None: ...
+ @overload
def approxQuantile(
- self, col: str, probabilities: List[float], relativeError: float
+ self,
+ col: str,
+ probabilities: Union[List[float], Tuple[float]],
+ relativeError: float,
) -> List[float]: ...
+ @overload
+ def approxQuantile(
+ self,
+ col: Union[List[str], Tuple[str]],
+ probabilities: Union[List[float], Tuple[float]],
+ relativeError: float,
+ ) -> List[List[float]]: ...
def corr(self, col1: str, col2: str, method: Optional[str] = ...) -> float: ...
def cov(self, col1: str, col2: str) -> float: ...
def crosstab(self, col1: str, col2: str) -> DataFrame: ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org