You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/23 22:19:13 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
xinrong-databricks opened a new pull request #33506:
URL: https://github.com/apache/spark/pull/33506
### What changes were proposed in this pull request?
Add set_categories to CategoricalAccessor and CategoricalIndex.
### Why are the changes needed?
set_categories is supported in pandas CategoricalAccessor and CategoricalIndex. We ought to follow pandas.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r676858391
##########
File path: python/pyspark/pandas/data_type_ops/categorical_ops.py
##########
@@ -42,6 +42,16 @@ def pretty_name(self) -> str:
def restore(self, col: pd.Series) -> pd.Series:
"""Restore column when to_pandas."""
+ try:
+ pd.Categorical.from_codes(
+ col.replace(np.nan, -1).astype(int),
+ categories=cast(CategoricalDtype, self.dtype).categories,
+ ordered=cast(CategoricalDtype, self.dtype).ordered,
+ )
+ except:
+ print(col)
+ print(self.dtype.categories)
+
Review comment:
Oh thanks, removed.
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
Review comment:
Modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886886821
**[Test build #141650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141650/testReport)** for PR 33506 at commit [`eaa58c0`](https://github.com/apache/spark/commit/eaa58c0618843d07f7877daa2ef84918eb6af8fd).
* This patch **fails Python style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887107505
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46183/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886998589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885976142
**[Test build #141591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141591/testReport)** for PR 33506 at commit [`6857523`](https://github.com/apache/spark/commit/68575238e6f7aea55dbc03799f61b50921847d02).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #33506:
URL: https://github.com/apache/spark/pull/33506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887054119
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886963605
**[Test build #141656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141656/testReport)** for PR 33506 at commit [`a22b990`](https://github.com/apache/spark/commit/a22b990d81792211b19b7cfcf4897ec265520138).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885955353
**[Test build #141587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141587/testReport)** for PR 33506 at commit [`bd5dc10`](https://github.com/apache/spark/commit/bd5dc104831c0c829aef3f8fdde16906ecfca394).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886997693
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885955008
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885959265
**[Test build #141588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141588/testReport)** for PR 33506 at commit [`69e4181`](https://github.com/apache/spark/commit/69e41811260e5ba7c16a9d41b136b8c9fc41ff30).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885979795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886961592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887092115
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141667/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885962980
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r676896916
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -680,12 +681,152 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
- ordered: bool = None,
+ new_categories: Union[pd.Index, List],
+ ordered: Optional[bool] = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+
+ Examples
+ --------
+ >>> s = ps.Series(list("abbccc"), dtype="category")
+ >>> s # doctest: +SKIP
+ 0 a
+ 1 b
+ 2 b
+ 3 c
+ 4 c
+ 5 c
+ dtype: category
+ Categories (3, object): ['a', 'b', 'c']
+
+ >>> s.cat.set_categories(['b', 'c']) # doctest: +SKIP
+ 0 NaN
+ 1 b
+ 2 b
+ 3 c
+ 4 c
+ 5 c
+ dtype: category
+ Categories (2, object): ['b', 'c']
+
+ >>> s.cat.set_categories([1, 2, 3], rename=True) # doctest: +SKIP
+ 0 1
+ 1 2
+ 2 2
+ 3 3
+ 4 3
+ 5 3
+ dtype: category
+ Categories (3, int64): [1, 2, 3]
+
+ >>> s.cat.set_categories([1, 2, 3], rename=True, ordered=True) # doctest: +SKIP
+ 0 1
+ 1 2
+ 2 2
+ 3 3
+ 4 3
+ 5 3
+ dtype: category
+ Categories (3, int64): [1 < 2 < 3]
+ """
+ from pyspark.pandas.frame import DataFrame
+
+ if not is_list_like(new_categories):
+ raise TypeError(
+ "Parameter 'new_categories' must be list-like, was '{}'".format(new_categories)
+ )
+
+ if ordered is None:
+ ordered = self.ordered
+
+ new_dtype = CategoricalDtype(new_categories, ordered=ordered)
+ scol = self._data.spark.column
+
+ if rename:
+ new_scol = (
+ F.when(scol >= len(new_categories), -1)
+ .otherwise(scol)
+ .alias(self._data._internal.data_spark_column_names[0])
+ )
+
+ internal = self._data._psdf._internal.with_new_spark_column(
+ self._data._column_label,
+ new_scol,
+ field=self._data._internal.data_fields[0].copy(
+ dtype=new_dtype, spark_type=IntegerType()
Review comment:
Why is this `IntegerType`?
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -668,6 +668,123 @@ def test_rename_categories(self):
lambda: psser.cat.rename_categories("x"),
)
+ def test_set_categories(self):
+ pdf, psdf = self.df_pair
+
+ pser = pdf.b
+ psser = psdf.b
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ )
Review comment:
I don't think we need tests with `pd.Index`.
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -668,6 +668,123 @@ def test_rename_categories(self):
lambda: psser.cat.rename_categories("x"),
)
+ def test_set_categories(self):
+ pdf, psdf = self.df_pair
+
+ pser = pdf.b
+ psser = psdf.b
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3])),
+ psser.cat.set_categories(pd.Index([0, 1, 3])),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4])),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4])),
+ )
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), rename=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3]), rename=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), rename=True),
+ )
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), ordered=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3]), ordered=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), ordered=True),
+ )
+
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), inplace=True, rename=True)
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), inplace=True, rename=True)
+ self.assert_eq(pser, psser)
+ self.assert_eq(pdf, psdf)
+
+ pser.cat.set_categories(pd.Index([2, 3, 1, 0]), inplace=True, rename=False),
+ psser.cat.set_categories(pd.Index([2, 3, 1, 0]), inplace=True, rename=False),
+ self.assert_eq(pser, psser)
+ self.assert_eq(pdf, psdf)
+
+ pdf, psdf = self.df_pair
+
+ pser = pdf.b
+ psser = psdf.b
+
+ self.assert_eq(
+ pser.cat.set_categories([0, 1, 3, 2]),
+ psser.cat.set_categories([0, 1, 3, 2]),
+ )
Review comment:
ditto for the following tests without `rename=True` argument.
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -668,6 +668,123 @@ def test_rename_categories(self):
lambda: psser.cat.rename_categories("x"),
)
+ def test_set_categories(self):
+ pdf, psdf = self.df_pair
+
+ pser = pdf.b
+ psser = psdf.b
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2])),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3])),
+ psser.cat.set_categories(pd.Index([0, 1, 3])),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4])),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4])),
+ )
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), rename=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3]), rename=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), rename=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), rename=True),
+ )
+
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), ordered=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3]), ordered=True),
+ )
+ self.assert_eq(
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), ordered=True),
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2, 4]), ordered=True),
+ )
+
+ pser.cat.set_categories(pd.Index([0, 1, 3, 2]), inplace=True, rename=True)
+ psser.cat.set_categories(pd.Index([0, 1, 3, 2]), inplace=True, rename=True)
+ self.assert_eq(pser, psser)
+ self.assert_eq(pdf, psdf)
+
+ pser.cat.set_categories(pd.Index([2, 3, 1, 0]), inplace=True, rename=False),
+ psser.cat.set_categories(pd.Index([2, 3, 1, 0]), inplace=True, rename=False),
+ self.assert_eq(pser, psser)
+ self.assert_eq(pdf, psdf)
+
+ pdf, psdf = self.df_pair
+
+ pser = pdf.b
+ psser = psdf.b
+
+ self.assert_eq(
+ pser.cat.set_categories([0, 1, 3, 2]),
+ psser.cat.set_categories([0, 1, 3, 2]),
+ )
Review comment:
Why do you use `[0, 1, 3, 2]` here? The result will always contain only `NaN`s.
##########
File path: python/pyspark/pandas/tests/indexes/test_category.py
##########
@@ -311,6 +311,55 @@ def test_rename_categories(self):
lambda: psidx.rename_categories("x"),
)
+ def test_set_categories(self):
+ pidx = pd.CategoricalIndex(["a", "b", "c", "d"])
+ psidx = ps.from_pandas(pidx)
+
+ self.assert_eq(
+ pidx.set_categories(pd.Index([0, 1, 3, 2])),
+ psidx.set_categories(pd.Index([0, 1, 3, 2])),
+ )
Review comment:
Shall we just use lists instead of `pd.Index`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887070787
**[Test build #141667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141667/testReport)** for PR 33506 at commit [`da64050`](https://github.com/apache/spark/commit/da640506d8f1155982490aa4945e2387791c992f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885968979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885966933
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46104/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885959275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141588/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885968979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887095049
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46183/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887009598
**[Test build #141660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141660/testReport)** for PR 33506 at commit [`2355ab5`](https://github.com/apache/spark/commit/2355ab559be9d4466e04956724c7b8c450f4cc72).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885959275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141588/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885962530
**[Test build #141587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141587/testReport)** for PR 33506 at commit [`bd5dc10`](https://github.com/apache/spark/commit/bd5dc104831c0c829aef3f8fdde16906ecfca394).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885986030
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887034374
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141660/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887112700
Thanks! merging to master/3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885978275
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46105/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r676986923
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -680,12 +681,152 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
- ordered: bool = None,
+ new_categories: Union[pd.Index, List],
+ ordered: Optional[bool] = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+
+ Examples
+ --------
+ >>> s = ps.Series(list("abbccc"), dtype="category")
+ >>> s # doctest: +SKIP
+ 0 a
+ 1 b
+ 2 b
+ 3 c
+ 4 c
+ 5 c
+ dtype: category
+ Categories (3, object): ['a', 'b', 'c']
+
+ >>> s.cat.set_categories(['b', 'c']) # doctest: +SKIP
+ 0 NaN
+ 1 b
+ 2 b
+ 3 c
+ 4 c
+ 5 c
+ dtype: category
+ Categories (2, object): ['b', 'c']
+
+ >>> s.cat.set_categories([1, 2, 3], rename=True) # doctest: +SKIP
+ 0 1
+ 1 2
+ 2 2
+ 3 3
+ 4 3
+ 5 3
+ dtype: category
+ Categories (3, int64): [1, 2, 3]
+
+ >>> s.cat.set_categories([1, 2, 3], rename=True, ordered=True) # doctest: +SKIP
+ 0 1
+ 1 2
+ 2 2
+ 3 3
+ 4 3
+ 5 3
+ dtype: category
+ Categories (3, int64): [1 < 2 < 3]
+ """
+ from pyspark.pandas.frame import DataFrame
+
+ if not is_list_like(new_categories):
+ raise TypeError(
+ "Parameter 'new_categories' must be list-like, was '{}'".format(new_categories)
+ )
+
+ if ordered is None:
+ ordered = self.ordered
+
+ new_dtype = CategoricalDtype(new_categories, ordered=ordered)
+ scol = self._data.spark.column
+
+ if rename:
+ new_scol = (
+ F.when(scol >= len(new_categories), -1)
+ .otherwise(scol)
+ .alias(self._data._internal.data_spark_column_names[0])
+ )
+
+ internal = self._data._psdf._internal.with_new_spark_column(
+ self._data._column_label,
+ new_scol,
+ field=self._data._internal.data_fields[0].copy(
+ dtype=new_dtype, spark_type=IntegerType()
Review comment:
cast to the original spark type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886922682
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885978055
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886884658
**[Test build #141650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141650/testReport)** for PR 33506 at commit [`eaa58c0`](https://github.com/apache/spark/commit/eaa58c0618843d07f7877daa2ef84918eb6af8fd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886886841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141650/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885955353
**[Test build #141587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141587/testReport)** for PR 33506 at commit [`bd5dc10`](https://github.com/apache/spark/commit/bd5dc104831c0c829aef3f8fdde16906ecfca394).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885959029
**[Test build #141588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141588/testReport)** for PR 33506 at commit [`69e4181`](https://github.com/apache/spark/commit/69e41811260e5ba7c16a9d41b136b8c9fc41ff30).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887034374
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141660/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885982286
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886889345
**[Test build #141654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141654/testReport)** for PR 33506 at commit [`5810cbd`](https://github.com/apache/spark/commit/5810cbd00c05470da9c6923c9fbef364ca27c8f4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886884658
**[Test build #141650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141650/testReport)** for PR 33506 at commit [`eaa58c0`](https://github.com/apache/spark/commit/eaa58c0618843d07f7877daa2ef84918eb6af8fd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885954610
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885969882
**[Test build #141591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141591/testReport)** for PR 33506 at commit [`6857523`](https://github.com/apache/spark/commit/68575238e6f7aea55dbc03799f61b50921847d02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886886841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141650/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886961592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887009598
**[Test build #141660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141660/testReport)** for PR 33506 at commit [`2355ab5`](https://github.com/apache/spark/commit/2355ab559be9d4466e04956724c7b8c450f4cc72).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r676860245
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
+ from pyspark.pandas.frame import DataFrame
+
+ if not is_list_like(new_categories):
+ raise TypeError(
+ "Parameter 'new_categories' must be list-like, was '{}'".format(new_categories)
+ )
+
+ if ordered is None:
+ ordered = self.ordered
+
+ new_dtype = CategoricalDtype(new_categories, ordered=ordered)
+ scol = self._data.spark.column
+
+ if rename:
+ new_scol = (
+ F.when(scol >= len(new_categories), -1)
+ .otherwise(scol)
+ .alias(name_like_string(self._data._column_label))
Review comment:
Makes sense, updated.
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
Review comment:
Certainly, added.
##########
File path: python/pyspark/pandas/indexes/category.py
##########
@@ -465,6 +465,70 @@ def rename_categories(
self.name
)
+ def set_categories(
+ self,
+ new_categories: Union[pd.Index, List],
+ ordered: bool = None,
+ rename: bool = False,
+ inplace: bool = False,
+ ) -> Optional["CategoricalIndex"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ CategoricalIndex with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887065859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887087004
**[Test build #141667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141667/testReport)** for PR 33506 at commit [`da64050`](https://github.com/apache/spark/commit/da640506d8f1155982490aa4945e2387791c992f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886990318
**[Test build #141654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141654/testReport)** for PR 33506 at commit [`5810cbd`](https://github.com/apache/spark/commit/5810cbd00c05470da9c6923c9fbef364ca27c8f4).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886956272
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887115657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46183/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887070787
**[Test build #141667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141667/testReport)** for PR 33506 at commit [`da64050`](https://github.com/apache/spark/commit/da640506d8f1155982490aa4945e2387791c992f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r675922174
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
Review comment:
Could you add some examples?
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
Review comment:
`Optional[bool]` ?
##########
File path: python/pyspark/pandas/indexes/category.py
##########
@@ -465,6 +465,70 @@ def rename_categories(
self.name
)
+ def set_categories(
+ self,
+ new_categories: Union[pd.Index, List],
+ ordered: bool = None,
+ rename: bool = False,
+ inplace: bool = False,
+ ) -> Optional["CategoricalIndex"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ CategoricalIndex with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
Review comment:
ditto.
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
rename: bool = False,
inplace: bool = False,
- ) -> "ps.Series":
- raise NotImplementedError()
+ ) -> Optional["ps.Series"]:
+ """
+ Set the categories to the specified new_categories.
+
+ `new_categories` can include new categories (which will result in
+ unused categories) or remove old categories (which results in values
+ set to NaN). If `rename==True`, the categories will simple be renamed
+ (less or more items than in old categories will result in values set to
+ NaN or in unused categories respectively).
+
+ This method can be used to perform more than one action of adding,
+ removing, and reordering simultaneously and is therefore faster than
+ performing the individual steps via the more specialised methods.
+
+ On the other hand this methods does not do checks (e.g., whether the
+ old categories are included in the new categories on a reorder), which
+ can result in surprising changes, for example when using special string
+ dtypes, which does not considers a S1 string equal to a single char
+ python string.
+
+ Parameters
+ ----------
+ new_categories : Index-like
+ The categories in new order.
+ ordered : bool, default False
+ Whether or not the categorical is treated as a ordered categorical.
+ If not given, do not change the ordered information.
+ rename : bool, default False
+ Whether or not the new_categories should be considered as a rename
+ of the old categories or as reordered categories.
+ inplace : bool, default False
+ Whether or not to reorder the categories in-place or return a copy
+ of this categorical with reordered categories.
+
+ Returns
+ -------
+ Series with reordered categories or None if inplace.
+
+ Raises
+ ------
+ ValueError
+ If new_categories does not validate as categories
+
+ See Also
+ --------
+ rename_categories : Rename categories.
+ reorder_categories : Reorder categories.
+ add_categories : Add new categories.
+ remove_categories : Remove the specified categories.
+ remove_unused_categories : Remove categories which are not used.
+ """
+ from pyspark.pandas.frame import DataFrame
+
+ if not is_list_like(new_categories):
+ raise TypeError(
+ "Parameter 'new_categories' must be list-like, was '{}'".format(new_categories)
+ )
+
+ if ordered is None:
+ ordered = self.ordered
+
+ new_dtype = CategoricalDtype(new_categories, ordered=ordered)
+ scol = self._data.spark.column
+
+ if rename:
+ new_scol = (
+ F.when(scol >= len(new_categories), -1)
+ .otherwise(scol)
+ .alias(name_like_string(self._data._column_label))
Review comment:
`self._data._internal.data_spark_column_name[0]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887025911
**[Test build #141660 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141660/testReport)** for PR 33506 at commit [`2355ab5`](https://github.com/apache/spark/commit/2355ab559be9d4466e04956724c7b8c450f4cc72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887092115
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141667/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887115657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46183/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887065859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885959029
**[Test build #141588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141588/testReport)** for PR 33506 at commit [`69e4181`](https://github.com/apache/spark/commit/69e41811260e5ba7c16a9d41b136b8c9fc41ff30).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885971479
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46105/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885979795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885955008
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885969882
**[Test build #141591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141591/testReport)** for PR 33506 at commit [`6857523`](https://github.com/apache/spark/commit/68575238e6f7aea55dbc03799f61b50921847d02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885986030
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886998589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886946868
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885941979
**[Test build #141586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141586/testReport)** for PR 33506 at commit [`dcce444`](https://github.com/apache/spark/commit/dcce444bf7e0994e7d6472a0e34ef48ea9569edb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r675920671
##########
File path: python/pyspark/pandas/data_type_ops/categorical_ops.py
##########
@@ -42,6 +42,16 @@ def pretty_name(self) -> str:
def restore(self, col: pd.Series) -> pd.Series:
"""Restore column when to_pandas."""
+ try:
+ pd.Categorical.from_codes(
+ col.replace(np.nan, -1).astype(int),
+ categories=cast(CategoricalDtype, self.dtype).categories,
+ ordered=cast(CategoricalDtype, self.dtype).ordered,
+ )
+ except:
+ print(col)
+ print(self.dtype.categories)
+
Review comment:
seems like a debug print?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886963605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885950093
**[Test build #141586 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141586/testReport)** for PR 33506 at commit [`dcce444`](https://github.com/apache/spark/commit/dcce444bf7e0994e7d6472a0e34ef48ea9569edb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886976451
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-886982679
**[Test build #141656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141656/testReport)** for PR 33506 at commit [`a22b990`](https://github.com/apache/spark/commit/a22b990d81792211b19b7cfcf4897ec265520138).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-885941979
**[Test build #141586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141586/testReport)** for PR 33506 at commit [`dcce444`](https://github.com/apache/spark/commit/dcce444bf7e0994e7d6472a0e34ef48ea9569edb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33506:
URL: https://github.com/apache/spark/pull/33506#issuecomment-887037243
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org