[spark] branch master updated: [SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f3e280b952d [SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`
f3e280b952d is described below

commit f3e280b952da8b8ab6c78371f3715cc674a73bc1
Author: Haejoon Lee <ha...@databricks.com>
AuthorDate: Thu Oct 19 13:57:01 2023 +0900

    [SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`
    
    ### What changes were proposed in this pull request?
    
    This PR proposes to deprecate `assertPandasOnSparkEqual`.
    
    ### Why are the changes needed?
    
    Now we have more pandas friendly testing utils such as `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal` and `ps.testing.assert_index_equal`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Not for now, but `assertPandasOnSparkEqual` will be removed in the future version.
    
    ### How was this patch tested?
    
    The existing CI should pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #43426 from itholic/SPARK-45553.
    
    Authored-by: Haejoon Lee <ha...@databricks.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/testing/pandasutils.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/python/pyspark/testing/pandasutils.py b/python/pyspark/testing/pandasutils.py
index bb9ca0dc74e..9abffefdbe7 100644
--- a/python/pyspark/testing/pandasutils.py
+++ b/python/pyspark/testing/pandasutils.py
@@ -341,6 +341,11 @@ def assertPandasOnSparkEqual(
 
     .. versionadded:: 3.5.0
 
+    .. deprecated:: 3.5.1
+        `assertPandasOnSparkEqual` will be removed in Spark 4.0.0.
+        Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal`
+        and `ps.testing.assert_index_equal` instead.
+
     Parameters
     ----------
     actual: pandas-on-Spark DataFrame, Series, or Index
@@ -393,6 +398,12 @@ def assertPandasOnSparkEqual(
     >>> s2 = ps.Index([212.3, 100.0001])
     >>> assertPandasOnSparkEqual(s1, s2, almost=True)  # pass, ps.Index obj are almost equal
     """
+    warnings.warn(
+        "`assertPandasOnSparkEqual` will be removed in Spark 4.0.0. "
+        "Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal` "
+        "and `ps.testing.assert_index_equal` instead.",
+        FutureWarning,
+    )
     if actual is None and expected is None:
         return True
     elif actual is None or expected is None:


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org