You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/10 06:43:53 UTC
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38579: [SPARK-40877][DOC][FOLLOW-UP] Update the doc of `DataFrame.stat.crosstab `
HyukjinKwon commented on code in PR #38579:
URL: https://github.com/apache/spark/pull/38579#discussion_r1018705370
##########
python/pyspark/sql/dataframe.py:
##########
@@ -4217,15 +4217,18 @@ def cov(self, col1: str, col2: str) -> float:
def crosstab(self, col1: str, col2: str) -> "DataFrame":
"""
Computes a pair-wise frequency table of the given columns. Also known as a contingency
- table. The number of distinct values for each column should be less than 1e4. At most 1e6
- non-zero pair frequencies will be returned.
Review Comment:
I think we can just remove this and don't add `.. versionchanged:: 3.4.0`. It's sort of improvement, and trivial to mention.
##########
python/pyspark/sql/dataframe.py:
##########
@@ -4217,15 +4217,18 @@ def cov(self, col1: str, col2: str) -> float:
def crosstab(self, col1: str, col2: str) -> "DataFrame":
"""
Computes a pair-wise frequency table of the given columns. Also known as a contingency
- table. The number of distinct values for each column should be less than 1e4. At most 1e6
- non-zero pair frequencies will be returned.
+ table.
The first column of each row will be the distinct values of `col1` and the column names
will be the distinct values of `col2`. The name of the first column will be `$col1_$col2`.
Pairs that have no occurrences will have zero as their counts.
:func:`DataFrame.crosstab` and :func:`DataFrameStatFunctions.crosstab` are aliases.
.. versionadded:: 1.4.0
+ .. versionchanged:: 3.4.0
+ The underlying implementation was optimized, then there is no limitaions on number of
+ distinct values or pairs.
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org