[GitHub] [spark] ueshin commented on a diff in pull request #41606: [SPARK-44061][PYTHON] Add assertDataFrameEqual util function

["ueshin (via GitHub)" <gi...@apache.org>]

ueshin commented on code in PR #41606:
URL: https://github.com/apache/spark/pull/41606#discussion_r1282615314


##########
python/pyspark/testing/utils.py:
##########
@@ -209,3 +219,200 @@ def check_error(
         self.assertEqual(
             expected, actual, f"Expected message parameters was '{expected}', got '{actual}'"
         )
+
+
+def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bool = False):
+    """
+    A util function to assert equality between DataFrames `df` and `expected`, with
+    optional parameter `check_row_order`.
+
+    .. versionadded:: 3.5.0
+
+    For float values, assert approximate equality (1e-5 by default).
+
+    Parameters
+    ----------
+    df : DataFrame
+        The DataFrame that is being compared or tested.
+
+    expected : DataFrame
+        The expected result of the operation, for comparison with the actual result.
+
+    check_row_order : bool, optional
+        A flag indicates whether the order of rows should be considered in the comparison.
+        If set to `False` (default), the row order is not taken into account.
+        If set to `True`, the order of rows is important and will be checked during comparison.
+
+    Examples
+    --------
+    >>> from pyspark.sql import SparkSession
+    >>> spark = SparkSession.builder.appName("assertDataFrameEqual example")\
+        .config("spark.some.config.option", "some-value").getOrCreate()
+    >>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
+    >>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
+    >>> assertDataFrameEqual(df1, df2) # pass
+    >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], \
+        schema=["id", "amount"])
+    >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], \
+        schema=["id", "amount"])
+    >>> assertDataFrameEqual(df1, df2) # fail  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+    ...
+    PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 0.66667 % )
+    [df]
+    Row(id='1', amount=1000.0)
+    <BLANKLINE>
+    [expected]
+    Row(id='1', amount=1001.0)
+    <BLANKLINE>
+    ********************
+    <BLANKLINE>
+    [df]
+    Row(id='3', amount=2000.0)
+    <BLANKLINE>
+    [expected]
+    Row(id='3', amount=2003.0)
+    <BLANKLINE>
+    ********************
+    <BLANKLINE>
+    <BLANKLINE>
+    """
+    if df is None and expected is None:
+        return True
+    elif df is None or expected is None:
+        return False

Review Comment:
   Some assertion error should be raised here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org