[spark] branch branch-3.2 updated: [SPARK-36370][PYTHON] _builtin_table directly imported from pandas instead of being redefined

[ue...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ueshin pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new e15daa3  [SPARK-36370][PYTHON] _builtin_table directly imported from pandas instead of being redefined
e15daa3 is described below

commit e15daa31b36669a7e29367e385f28b6ba25acf09
Author: Cedric-Magnan <ce...@artefact.com>
AuthorDate: Tue Aug 17 10:46:49 2021 -0700

    [SPARK-36370][PYTHON] _builtin_table directly imported from pandas instead of being redefined
    
    ### What changes were proposed in this pull request?
    Suggesting to refactor the way the _builtin_table is defined in the `python/pyspark/pandas/groupby.py` module.
    Pandas has recently refactored the way we import the _builtin_table and is now part of the pandas.core.common module instead of being an attribute of the pandas.core.base.SelectionMixin class.
    
    ### Why are the changes needed?
    This change is not fully needed but the current implementation redefines this table within pyspark, so any changes of this table from the pandas library would need to be updated in the pyspark repository as well.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Ran the following command successfully :
    ```sh
    python/run-tests --testnames 'pyspark.pandas.tests.test_groupby'
    ```
    Tests passed in 327 seconds
    
    Closes #33687 from Cedric-Magnan/_builtin_table_from_pandas.
    
    Authored-by: Cedric-Magnan <ce...@artefact.com>
    Signed-off-by: Takuya UESHIN <ue...@databricks.com>
    (cherry picked from commit 964dfe254ff8ebf9d7f5c7115ff8f79da3f28261)
    Signed-off-by: Takuya UESHIN <ue...@databricks.com>
---
 python/pyspark/pandas/groupby.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/python/pyspark/pandas/groupby.py b/python/pyspark/pandas/groupby.py
index 376592d..2daf80f 100644
--- a/python/pyspark/pandas/groupby.py
+++ b/python/pyspark/pandas/groupby.py
@@ -20,13 +20,13 @@ A wrapper for GroupedData to behave similar to pandas GroupBy.
 """
 
 from abc import ABCMeta, abstractmethod
-import builtins
 import sys
 import inspect
 from collections import OrderedDict, namedtuple
 from distutils.version import LooseVersion
 from functools import partial
 from itertools import product
+from pkg_resources import parse_version  # type: ignore
 from typing import (
     Any,
     Callable,
@@ -44,10 +44,16 @@ from typing import (
     TYPE_CHECKING,
 )
 
-import numpy as np
 import pandas as pd
 from pandas.api.types import is_hashable, is_list_like
 
+if parse_version(pd.__version__) >= parse_version("1.3.0"):
+    from pandas.core.common import _builtin_table
+else:
+    from pandas.core.base import SelectionMixin
+
+    _builtin_table = SelectionMixin._builtin_table
+
 from pyspark.sql import Column, DataFrame as SparkDataFrame, Window, functions as F
 from pyspark.sql.types import (  # noqa: F401
     DataType,
@@ -97,12 +103,6 @@ if TYPE_CHECKING:
 # to keep it the same as pandas
 NamedAgg = namedtuple("NamedAgg", ["column", "aggfunc"])
 
-_builtin_table = {
-    builtins.sum: np.sum,
-    builtins.max: np.max,
-    builtins.min: np.min,
-}  # type: Dict[Callable, Callable]
-
 
 class GroupBy(Generic[FrameLike], metaclass=ABCMeta):
     """

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org