You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/05 04:43:03 UTC
[GitHub] [spark] dchvn opened a new pull request #34179: [SPARK-36930][PYSPARK] Add dtypes for MultiIndex
dchvn opened a new pull request #34179:
URL: https://github.com/apache/spark/pull/34179
### What changes were proposed in this pull request?
Add dtypes for MultiIndex
### Why are the changes needed?
Add dtypes for MultiIndex
### Does this PR introduce _any_ user-facing change?
After this PR user can use ```MultiIndex.dtypes``` for:
``` python
>>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 5, 6, 7, 8, 9]], names=("zero", "one"))
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... index=idx,
... )
>>> psdf = ps.from_pandas(pdf)
>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, pyspark.pandas.typedef.typehints.IndexNameType, pyspark.pandas.typedef.typehints.NameType, pyspark.pandas.typedef.typehints.NameType]
```
### How was this patch tested?
unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934479912
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934109733
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722841375
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,13 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> pd.Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=[name if len(name) > 1 else name[0] for name in self._internal.index_names],
Review comment:
can we add a test case with multiple labels too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934146175
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934529163
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935587613
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722847893
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
> hmm, if we return `ps.Series`, we can not use index type hint for now. Can we use `pd.Series` ?
>
> ```python
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
> return create_tuple_for_frame_type(params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
> return Tuple[_extract_types(params)]
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
> index_params = _to_tuple_of_params(index_params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
> params = tuple(params)
> File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
> return MissingPandasLikeSeries.__iter__(self)
> File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
> raise PandasNotImplementedError(
> pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
> ```
Yeah, I think it's normal because we don't support the iteration for `Series` yet.
I wonder if we can say the nlevels of MultiIndex is always expected to be small.
If the nlevels of MultiIndex is always expected to be small, it's okay to just return the pandas Series.
Otherwise, we should return pandas-on-Spark Series.
I'm pretty sure most of them expected to be small,
but could you double check if it's okay, @HyukjinKwon @ueshin @xinrong-databricks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934112177
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143832/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934099291
**[Test build #143832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143832/testReport)** for PR 34179 at commit [`44a8349`](https://github.com/apache/spark/commit/44a8349ee0b6d5dc83655e8bb40033b70c1f6983).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r723803580
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
Yeah, I will make a PR soon, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935392853
**[Test build #143869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143869/testReport)** for PR 34179 at commit [`d92d8cb`](https://github.com/apache/spark/commit/d92d8cb6312edb181474f16bc29c9ee6139c6719).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934525554
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143857/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934180529
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935576303
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934242317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722847893
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
> hmm, if we return `ps.Series`, we can not use index type hint for now. Can we use `pd.Series` ?
>
> ```python
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
> return create_tuple_for_frame_type(params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
> return Tuple[_extract_types(params)]
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
> index_params = _to_tuple_of_params(index_params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
> params = tuple(params)
> File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
> return MissingPandasLikeSeries.__iter__(self)
> File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
> raise PandasNotImplementedError(
> pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
> ```
Yeah, I think it's normal because we don't support the iteration for `Series` yet.
I wonder if we can say the nlevels of MultiIndex is always expected to be small.
If the nlevels of MultiIndex is always expected to be small, it's okay to just return the pandas Series.
Otherwise, we should return pandas-on-Spark Series.
WDYT, @HyukjinKwon @ueshin @xinrong-databricks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935328969
LGTM if the remaining comments are resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935487219
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935438473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143869/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r723793429
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
@dchvn, `dtypes` in `MultiIndex` only exists in pandas 1.3+ so the tests won't be able to run with lower pandas versions. Can you make a followup PR to skip conditionally this test when pandas version is lower?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934138822
**[Test build #143839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143839/testReport)** for PR 34179 at commit [`ed23a01`](https://github.com/apache/spark/commit/ed23a01868293abfa5130d70fe22065452635847).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934083911
**[Test build #143832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143832/testReport)** for PR 34179 at commit [`44a8349`](https://github.com/apache/spark/commit/44a8349ee0b6d5dc83655e8bb40033b70c1f6983).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934149536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r721920147
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,14 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ idx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ pdf = pd.DataFrame(np.random.randn(4, 5), idx)
+ psdf = ps.from_pandas(pdf)
+
+ self.assert_eq(psdf.index.dtypes, pdf.index.dtypes)
Review comment:
nit: Can we just compare with MultiIndex itself ??
For example,
```python
arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
psmidx = ps.from_pandas(pmidx)
self.assert_eq(psmidx.dtypes, pmidx.dtypes)
```
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
Can we create pandas-on-Spark Series rather than pandas Series ??
For example,
```python
return ps.Series(
[field.dtype.name for field in self._internal.index_fields],
index=[name if len(name) > 1 else name[0] for name in self._internal.index_names])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r723793429
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
@dchvn, `dtypes` in `MultiIndex` only exists in pandas 1.3+ so the tests won't be able to run with lower pandas versions. Can you make a followup PR to skip conditionally this test when pandas version is lower?
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
Also feel free to make a separate PR to skip this in doctest:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/typedef/typehints.py#L728-L729
by changing:
```diff
>>> ps.DataFrame[zip(pdf.index.names, pdf.index.dtypes), zip(pdf.columns, pdf.dtypes)]
- ... # doctest: +ELLIPSIS
+ ... # doctest: +ELLIPSIS, +SKIP
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722841818
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,13 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> pd.Series:
Review comment:
Let's also add it in docs https://github.com/apache/spark/blob/master/python/docs/source/reference/pyspark.pandas/indexing.rst#multiindex-properties.
And add some docstrings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722847893
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
> hmm, if we return `ps.Series`, we can not use index type hint for now. Can we use `pd.Series` ?
>
> ```python
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
> return create_tuple_for_frame_type(params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
> return Tuple[_extract_types(params)]
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
> index_params = _to_tuple_of_params(index_params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
> params = tuple(params)
> File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
> return MissingPandasLikeSeries.__iter__(self)
> File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
> raise PandasNotImplementedError(
> pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
> ```
Yeah, I think it's normal because we don't support the iteration for `Series` yet.
I wonder if we can say the length of MultiIndex is always expected to be small.
If the length of MultiIndex is always expected to be small, it's okay to just return the pandas Series.
Otherwise, we should return pandas-on-Spark Series.
WDYT, @HyukjinKwon @ueshin @xinrong-databricks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935561695
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934214069
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934529163
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934242317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48352/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34179:
URL: https://github.com/apache/spark/pull/34179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935438473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143869/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934475709
**[Test build #143857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143857/testReport)** for PR 34179 at commit [`d6d489a`](https://github.com/apache/spark/commit/d6d489a61bcceee152be7769dbb0784a599d01a2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r721939348
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,14 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ idx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ pdf = pd.DataFrame(np.random.randn(4, 5), idx)
+ psdf = ps.from_pandas(pdf)
+
+ self.assert_eq(psdf.index.dtypes, pdf.index.dtypes)
Review comment:
thanks, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934525554
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143857/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722838976
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
That;s fine. I think this is matched with DataFrame.dtypes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935422309
**[Test build #143869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143869/testReport)** for PR 34179 at commit [`d92d8cb`](https://github.com/apache/spark/commit/d92d8cb6312edb181474f16bc29c9ee6139c6719).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934149536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934112177
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143832/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722274565
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
I reverted to return ```pd.Series```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934119998
**[Test build #143839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143839/testReport)** for PR 34179 at commit [`ed23a01`](https://github.com/apache/spark/commit/ed23a01868293abfa5130d70fe22065452635847).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935587613
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722876518
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,13 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> pd.Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=[name if len(name) > 1 else name[0] for name in self._internal.index_names],
Review comment:
thanks, updated
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,13 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> pd.Series:
Review comment:
updated
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,13 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
Review comment:
thanks, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34179: [SPARK-36930][PYSPARK] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934056247
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935328120
Could you add an doc reference in the `MultiIndex Properties` under `docs/source/reference/pyspark.pandas/indexing.rst` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934056247
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935392853
**[Test build #143869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143869/testReport)** for PR 34179 at commit [`d92d8cb`](https://github.com/apache/spark/commit/d92d8cb6312edb181474f16bc29c9ee6139c6719).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722854286
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
Yeah, I think it's fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934083911
**[Test build #143832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143832/testReport)** for PR 34179 at commit [`44a8349`](https://github.com/apache/spark/commit/44a8349ee0b6d5dc83655e8bb40033b70c1f6983).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r721943317
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
hmm, if we return ```ps.Series```, we can not use index type hint for now. Can we use ```pd.Series``` ?
``` python
>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
return create_tuple_for_frame_type(params)
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
return Tuple[_extract_types(params)]
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
index_params = _to_tuple_of_params(index_params)
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
params = tuple(params)
File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
return MissingPandasLikeSeries.__iter__(self)
File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
raise PandasNotImplementedError(
pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34179: [SPARK-36930][PYSPARK] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934057987
cc @HyukjinKwon , Do you like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934475709
**[Test build #143857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143857/testReport)** for PR 34179 at commit [`d6d489a`](https://github.com/apache/spark/commit/d6d489a61bcceee152be7769dbb0784a599d01a2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934083415
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r721931454
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
Thank you, I will update. I copied behavior of ps.DataFrame.dtypes. Should we change it too?
https://github.com/apache/spark/blob/d6786e036d610476a3be0fca5b16ba819dcbc013/python/pyspark/pandas/frame.py#L6276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722841974
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,13 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
Review comment:
Let's start following PySpark's convention now since we finished initial porting. Can you add a comment with JIRA prefix? See also https://spark.apache.org/contributing.html
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,13 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> pd.Series:
Review comment:
Let;s also add versionadded directive in docstring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935299674
cc @xinrong-databricks @ueshin FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic edited a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935328120
Could you add an doc reference in the `MultiIndex Properties` under `docs/source/reference/pyspark.pandas/indexing.rst` ? (It's same as https://github.com/apache/spark/pull/34179#discussion_r722841818)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic edited a comment on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935328120
Could you add an doc reference in the `MultiIndex Properties` under `docs/source/reference/pyspark.pandas/indexing.rst` ? (It's a part of https://github.com/apache/spark/pull/34179#discussion_r722841818)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934529112
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934083584
cc @ueshin @itholic @xinrong-databricks FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r721943317
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
hmm, if we return ```ps.Series```, we can not use index type hint for now. Can we use ```pd.Series```
``` python
>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
return create_tuple_for_frame_type(params)
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
return Tuple[_extract_types(params)]
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
index_params = _to_tuple_of_params(index_params)
File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
params = tuple(params)
File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
return MissingPandasLikeSeries.__iter__(self)
File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
raise PandasNotImplementedError(
pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934119998
**[Test build #143839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143839/testReport)** for PR 34179 at commit [`ed23a01`](https://github.com/apache/spark/commit/ed23a01868293abfa5130d70fe22065452635847).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-934497744
**[Test build #143857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143857/testReport)** for PR 34179 at commit [`d6d489a`](https://github.com/apache/spark/commit/d6d489a61bcceee152be7769dbb0784a599d01a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722847893
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex", property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in self._internal.index_names]
+ ),
+ )
Review comment:
> hmm, if we return `ps.Series`, we can not use index type hint for now. Can we use `pd.Series` ?
>
> ```python
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in __class_getitem__
> return create_tuple_for_frame_type(params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729, in create_tuple_for_frame_type
> return Tuple[_extract_types(params)]
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760, in _extract_types
> index_params = _to_tuple_of_params(index_params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833, in _to_tuple_of_params
> params = tuple(params)
> File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
> return MissingPandasLikeSeries.__iter__(self)
> File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in unsupported_function
> raise PandasNotImplementedError(
> pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
> ```
Yeah, I think it's normal because we don't support the iteration for `Series` yet.
I wonder if we can say the nlevels of MultiIndex is always expected to be small.
If the nlevels of MultiIndex is always expected to be small, it's okay to just return the pandas Series.
Otherwise, we should return pandas-on-Spark Series.
I'm pretty sure most of them will be small,
but could you double check if it's okay, @HyukjinKwon @ueshin @xinrong-databricks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34179: [SPARK-36930][PYTHON] Add dtypes for MultiIndex
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935299674
cc @xinrong-databricks @ueshin @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34179:
URL: https://github.com/apache/spark/pull/34179#issuecomment-935628081
Thank you @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r723794420
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
Also feel free to make a separate PR to skip this in doctest:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/typedef/typehints.py#L728-L729
by changing:
```diff
>>> ps.DataFrame[zip(pdf.index.names, pdf.index.dtypes), zip(pdf.columns, pdf.dtypes)]
- ... # doctest: +ELLIPSIS
+ ... # doctest: +ELLIPSIS, +SKIP
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34179: [SPARK-36930][PYTHON] Support ps.MultiIndex.dtypes
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r723803580
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -6000,6 +6000,20 @@ def test_combine_first(self):
expected_pdf = pd.DataFrame({"A": [None, 0], "B": [4.0, 1.0], "C": [3, 3]})
self.assert_eq(expected_pdf, psdf1.combine_first(psdf2))
+ def test_multi_index_dtypes(self):
+ # SPARK-36930: Support ps.MultiIndex.dtypes
+ arrays = [[1, 1, 2, 2], ["red", "blue", "red", "blue"]]
+ pmidx = pd.MultiIndex.from_arrays(arrays, names=("number", "color"))
+ psmidx = ps.from_pandas(pmidx)
+
+ self.assert_eq(psmidx.dtypes, pmidx.dtypes)
Review comment:
Yeah, I will make a PR soon, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org