You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/06 12:22:26 UTC
[GitHub] [spark] beliefer opened a new pull request, #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
beliefer opened a new pull request, #38938:
URL: https://github.com/apache/spark/pull/38938
### What changes were proposed in this pull request?
Implement `DataFrame.describe` with a proto message
### Why are the changes needed?
for Connect API coverage
### Does this PR introduce _any_ user-facing change?
'No'. New API
### How was this patch tested?
New test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #38938:
URL: https://github.com/apache/spark/pull/38938#issuecomment-1339800277
Some suggestions after check the failed CI job:
1. you can run `dev/reformat-python` to format your python code.
2. I left some ideas for how to develop for python side that might help: https://github.com/apache/spark/pull/38899#issuecomment-1336537633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041676903
##########
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -404,6 +405,18 @@ message StatSummary {
repeated string statistics = 2;
}
+// Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
+// and max. If no columns are given, this function computes statistics for all numerical or
+// string columns.
+// It will invoke 'Dataset.describe' to compute the results.
Review Comment:
it has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38938: [SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38938:
URL: https://github.com/apache/spark/pull/38938#issuecomment-1340575126
merged into master, thank you for working on this @beliefer !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041729116
##########
python/pyspark/sql/connect/dataframe.py:
##########
@@ -1239,6 +1239,16 @@ def summary(self, *statistics: str) -> "DataFrame":
session=self._session,
)
+ def describe(self, *cols: str) -> "DataFrame":
+ _cols: List[str] = list(cols)
Review Comment:
I will update it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #38938: [SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #38938: [SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
URL: https://github.com/apache/spark/pull/38938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041708865
##########
python/pyspark/sql/connect/dataframe.py:
##########
@@ -1239,6 +1239,16 @@ def summary(self, *statistics: str) -> "DataFrame":
session=self._session,
)
+ def describe(self, *cols: str) -> "DataFrame":
+ _cols: List[str] = list(cols)
Review Comment:
oops I don't know why they are gone but we need those new lines. Otherwise this PR won't pass the lint check...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #38938: [SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #38938:
URL: https://github.com/apache/spark/pull/38938#issuecomment-1340585400
@amaliujia @zhengruifeng @HyukjinKwon Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041665659
##########
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -404,6 +405,18 @@ message StatSummary {
repeated string statistics = 2;
}
+// Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
+// and max. If no columns are given, this function computes statistics for all numerical or
+// string columns.
+// It will invoke 'Dataset.describe' to compute the results.
+message StatDescribe {
+ // (Required) The input relation.
+ Relation input = 1;
+
+ // Columns to compute statistics on.
+ repeated string cols = 2;
Review Comment:
I got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #38938:
URL: https://github.com/apache/spark/pull/38938#issuecomment-1340315896
Not sure why in the suggestion those newlines were gone. But we need those new lines. Otherwise this PR won't pass the lint check...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041314131
##########
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -404,6 +405,18 @@ message StatSummary {
repeated string statistics = 2;
}
+// Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
+// and max. If no columns are given, this function computes statistics for all numerical or
+// string columns.
+// It will invoke 'Dataset.describe' to compute the results.
+message StatDescribe {
+ // (Required) The input relation.
+ Relation input = 1;
+
+ // Columns to compute statistics on.
+ repeated string cols = 2;
Review Comment:
Can you follow https://github.com/apache/spark/blob/master/connector/connect/docs/adding-proto-messages.md to mark if this is required or optional?
##########
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -404,6 +405,18 @@ message StatSummary {
repeated string statistics = 2;
}
+// Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
+// and max. If no columns are given, this function computes statistics for all numerical or
+// string columns.
+// It will invoke 'Dataset.describe' to compute the results.
Review Comment:
`It will invoke 'Dataset.describe' to compute the results.`
I tend to avoid say this as this exposes implementation details while the comment here actually serves as proto documentation. (E.g. we might switch the implementation to something else but forget to update here thus cause inconsistency)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041691534
##########
python/pyspark/sql/connect/dataframe.py:
##########
@@ -1239,6 +1239,16 @@ def summary(self, *statistics: str) -> "DataFrame":
session=self._session,
)
+ def describe(self, *cols: str) -> "DataFrame":
+ _cols: List[str] = list(cols)
Review Comment:
```suggestion
"""Computes basic statistics for numeric and string columns
.. versionadded:: 3.4.0
This include count, mean, stddev, min, and max. If no columns are
given, this function computes statistics for all numerical or string columns.
Notes
-----
This function is meant for exploratory data analysis, as we make no
guarantee about the backward compatibility of the schema of the resulting
:class:`DataFrame`.
Use summary for expanded statistics and control over which statistics to compute.
Parameters
----------
cols : str, list, optional
Column name or list of column names to describe by (default All columns).
Returns
-------
:class:`DataFrame`
A new DataFrame that describes (provides statistics) given DataFrame.
"""
_cols: List[str] = list(cols)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38938: [WIP][SPARK-41403][CONNECT][PYTHON] Implement `DataFrame.describe`
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38938:
URL: https://github.com/apache/spark/pull/38938#discussion_r1041691534
##########
python/pyspark/sql/connect/dataframe.py:
##########
@@ -1239,6 +1239,16 @@ def summary(self, *statistics: str) -> "DataFrame":
session=self._session,
)
+ def describe(self, *cols: str) -> "DataFrame":
+ _cols: List[str] = list(cols)
Review Comment:
```suggestion
"""Computes basic statistics for numeric and string columns.
.. versionadded:: 3.4.0
This include count, mean, stddev, min, and max. If no columns are
given, this function computes statistics for all numerical or string columns.
Notes
-----
This function is meant for exploratory data analysis, as we make no
guarantee about the backward compatibility of the schema of the resulting
:class:`DataFrame`.
Use summary for expanded statistics and control over which statistics to compute.
Parameters
----------
cols : str, list, optional
Column name or list of column names to describe by (default All columns).
Returns
-------
:class:`DataFrame`
A new DataFrame that describes (provides statistics) given DataFrame.
"""
_cols: List[str] = list(cols)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org