You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/26 18:49:44 UTC
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33506: [SPARK-36260][PYTHON] Add set_categories to CategoricalAccessor and CategoricalIndex
xinrong-databricks commented on a change in pull request #33506:
URL: https://github.com/apache/spark/pull/33506#discussion_r676858391
##########
File path: python/pyspark/pandas/data_type_ops/categorical_ops.py
##########
@@ -42,6 +42,16 @@ def pretty_name(self) -> str:
def restore(self, col: pd.Series) -> pd.Series:
"""Restore column when to_pandas."""
+ try:
+ pd.Categorical.from_codes(
+ col.replace(np.nan, -1).astype(int),
+ categories=cast(CategoricalDtype, self.dtype).categories,
+ ordered=cast(CategoricalDtype, self.dtype).ordered,
+ )
+ except:
+ print(col)
+ print(self.dtype.categories)
+
Review comment:
Oh thanks, removed.
##########
File path: python/pyspark/pandas/categorical.py
##########
@@ -593,12 +595,107 @@ def reorder_categories(
def set_categories(
self,
- new_categories: pd.Index,
+ new_categories: Union[pd.Index, List],
ordered: bool = None,
Review comment:
Modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org