You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "asl3 (via GitHub)" <gi...@apache.org> on 2023/07/11 23:05:45 UTC
[GitHub] [spark] asl3 opened a new pull request, #41947: [SPARK-44217] [PYTHON] Allow custom precision for fp approx equality
asl3 opened a new pull request, #41947:
URL: https://github.com/apache/spark/pull/41947
### What changes were proposed in this pull request?
This PR adds an optional argument, `precision`, to `assertDataFrameEqual` to allow users to customize precision level when asserting approximate equality for float values in DataFrames.
### Why are the changes needed?
The change adds additional flexibility and user customization to the `assertDataFrameEqual` util.
### Does this PR introduce _any_ user-facing change?
Yes, the PR changes the args accepted for the user-facing util, `assertDataFrameEqual`.
### How was this patch tested?
Added tests to `runtime/python/pyspark/sql/tests/test_utils.py` and `runtime/python/pyspark/sql/tests/connect/test_utils.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1262424104
##########
python/pyspark/testing/utils.py:
##########
@@ -250,32 +268,29 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
.config("spark.some.config.option", "some-value").getOrCreate()
Review Comment:
Can we remove:
```
>>> spark = SparkSession.builder.appName("assertDataFrameEqual example")\
.config("spark.some.config.option", "some-value").getOrCreate()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1261546994
##########
python/pyspark/testing/utils.py:
##########
@@ -234,14 +240,24 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
----------
df : DataFrame
The DataFrame that is being compared or tested.
-
expected : DataFrame
The expected result of the operation, for comparison with the actual result.
-
check_row_order : bool, optional
A flag indicating whether the order of rows should be considered in the comparison.
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ rtol : float, optional
+ The relative tolerance, used in asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default. (See Notes)
+ atol : float, optional
Review Comment:
Let's also add an example for these?
##########
python/pyspark/sql/tests/test_utils.py:
##########
@@ -115,6 +115,88 @@ def test_assert_approx_equal_arraytype_float(self):
assertDataFrameEqual(df1, df2)
+ def test_assert_approx_equal_arraytype_float_default_rtol_fail(self):
+ # fails with default rtol, 1e-5
+ df1 = self.spark.createDataFrame(
+ data=[
+ ("student1", [97.01, 89.23]),
+ ("student2", [91.86, 84.34]),
+ ],
+ schema=StructType(
+ [
+ StructField("student", StringType(), True),
+ StructField("grades", ArrayType(FloatType()), True),
+ ]
+ ),
+ )
+ df2 = self.spark.createDataFrame(
+ data=[
+ ("student1", [97.01, 89.23]),
+ ("student2", [91.86, 84.341]),
+ ],
+ schema=StructType(
+ [
+ StructField("student", StringType(), True),
+ StructField("grades", ArrayType(FloatType()), True),
+ ]
+ ),
+ )
+
+ expected_error_message = "Results do not match: "
+ percent_diff = (1 / 2) * 100
+ expected_error_message += "( %.5f %% )" % percent_diff
+ diff_msg = (
+ "[df]"
+ + "\n"
+ + str(df1.collect()[1])
+ + "\n\n"
+ + "[expected]"
+ + "\n"
+ + str(df2.collect()[1])
+ + "\n\n"
+ + "********************"
+ + "\n\n"
+ )
+ expected_error_message += "\n" + diff_msg
+
+ with self.assertRaises(PySparkAssertionError) as pe:
+ assertDataFrameEqual(df1, df2)
+
+ self.check_error(
+ exception=pe.exception,
+ error_class="DIFFERENT_ROWS",
+ message_parameters={"error_msg": expected_error_message},
+ )
+
+ def test_assert_approx_equal_arraytype_float_custom_rtol_pass(self):
+ # passes with custom rtol, 1e-2
+ df1 = self.spark.createDataFrame(
+ data=[
+ ("student1", [97.01, 89.23]),
+ ("student2", [91.86, 84.34]),
+ ],
+ schema=StructType(
+ [
+ StructField("student", StringType(), True),
+ StructField("grades", ArrayType(FloatType()), True),
Review Comment:
Can we also test 1) DoubleType 2) DecimalType?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1261940687
##########
python/pyspark/testing/utils.py:
##########
@@ -251,31 +267,35 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
>>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2) # pass
+ >>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2, rtol=1e-1) # pass
>>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], \
schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], \
schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # fail # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> assertDataFrameEqual(df1, df2)
+ ... # doctest: +IGNORE_EXCEPTION_DETAIL, +NORMALIZE_WHITESPACE
Review Comment:
We already have `NORMALIZE_WHITESPACE` at the bottom so we don't need `NORMALIZE_WHITESPACE` acutally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1263172651
##########
python/pyspark/testing/utils.py:
##########
@@ -336,7 +353,7 @@ def compare_vals(val1, val2):
and all(compare_vals(val1[k], val2[k]) for k in val1.keys())
)
elif isinstance(val1, float) and isinstance(val2, float):
- if abs(val1 - val2) > 1e-5:
Review Comment:
should we also take `decimal` into account?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41947:
URL: https://github.com/apache/spark/pull/41947#issuecomment-1635329507
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
URL: https://github.com/apache/spark/pull/41947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1260511824
##########
python/pyspark/testing/utils.py:
##########
@@ -243,6 +245,10 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ precision : float, optional
+ The level of precision when asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default.
Review Comment:
good point, we actually use `rtol` and `atol` in both scala and python:
in python:
https://github.com/apache/spark/blob/5a702f22f49ca6a1b6220ac645e3fce70ec5189d/python/pyspark/ml/tests/test_algorithms.py#L390-L396
https://github.com/apache/spark/blob/f3edc0c3570ea4b28f68607d6404c6b8d0801161/python/pyspark/pandas/tests/plot/test_series_plot_matplotlib.py#L364-L367
in scala:
https://github.com/apache/spark/blob/d679dabdd1b5ad04b8c7deb1c06ce886a154a928/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala#L379-L400
https://github.com/apache/spark/blob/d679dabdd1b5ad04b8c7deb1c06ce886a154a928/mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala#L1021-L1040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217] [PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1260417791
##########
python/pyspark/testing/utils.py:
##########
@@ -243,6 +245,10 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
Review Comment:
I think you can remove whitespaces between each parameter (https://numpydoc.readthedocs.io/en/latest/format.html#parameters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1263172651
##########
python/pyspark/testing/utils.py:
##########
@@ -336,7 +353,7 @@ def compare_vals(val1, val2):
and all(compare_vals(val1[k], val2[k]) for k in val1.keys())
)
elif isinstance(val1, float) and isinstance(val2, float):
- if abs(val1 - val2) > 1e-5:
Review Comment:
~~ should we also take `decimal` into account? ~~
##########
python/pyspark/testing/utils.py:
##########
@@ -336,7 +353,7 @@ def compare_vals(val1, val2):
and all(compare_vals(val1[k], val2[k]) for k in val1.keys())
)
elif isinstance(val1, float) and isinstance(val2, float):
- if abs(val1 - val2) > 1e-5:
Review Comment:
~~should we also take `decimal` into account?~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1262425446
##########
python/pyspark/testing/utils.py:
##########
@@ -250,32 +268,29 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
.config("spark.some.config.option", "some-value").getOrCreate()
>>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # pass
- >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], \
- schema=["id", "amount"])
- >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], \
- schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # fail # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> assertDataFrameEqual(df1, df2) # DataFrames are equal
+ >>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2, rtol=1e-1) # DataFrames are approx equal by rtol
+ >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)],
+ ... schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)],
+ ... schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
Review Comment:
Can we fix as follows?
```suggestion
>>> assertDataFrameEqual(df1, df2) # DataFrames are equal
Explanation
>>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2, rtol=1e-1) #
Explanation
>>> df1 = spark.createDataFrame(
... data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(
... data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1261940266
##########
python/pyspark/testing/utils.py:
##########
@@ -221,7 +221,13 @@ def check_error(
)
-def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bool = False):
+def assertDataFrameEqual(
+ df: DataFrame,
+ expected: DataFrame,
+ check_row_order: bool = False,
Review Comment:
While we're here, I think we should remove `check_row_order`. We don't really guarantee the order in Spark Dataframe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asl3 commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "asl3 (via GitHub)" <gi...@apache.org>.
asl3 commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1262060087
##########
python/pyspark/testing/utils.py:
##########
@@ -221,7 +221,13 @@ def check_error(
)
-def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bool = False):
+def assertDataFrameEqual(
+ df: DataFrame,
+ expected: DataFrame,
+ check_row_order: bool = False,
Review Comment:
we decided to keep (renamed to `checkRowOrder`), with added documentation clarifying that ordering isn't guaranteed unless df is explicitly sorted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217] [PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1260418104
##########
python/pyspark/testing/utils.py:
##########
@@ -243,6 +245,10 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ precision : float, optional
+ The level of precision when asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default.
+
Examples
--------
>>> from pyspark.sql import SparkSession
Review Comment:
Remove ` <BLANKLINE>`s but add `+NORMALIZE_WHITESPACE` for doctest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217] [PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1260419611
##########
python/pyspark/testing/utils.py:
##########
@@ -243,6 +245,10 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ precision : float, optional
+ The level of precision when asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default.
Review Comment:
Maybe it's okay for now. but I think you should probably take a look at other libraries such as NumPy's allclose `rtol` and `atol` https://numpy.org/doc/stable/reference/generated/numpy.allclose.html#numpy-allclose. I think the name has to better be tolerance but don't have a strong preference. cc @zhengruifeng and @WeichenXu123 in case you guys have some feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1260512556
##########
python/pyspark/testing/utils.py:
##########
@@ -243,6 +245,10 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ precision : float, optional
+ The level of precision when asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default.
Review Comment:
so I think we'd better follow `atol` & `rtol` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1261940449
##########
python/pyspark/testing/utils.py:
##########
@@ -234,14 +240,24 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
----------
df : DataFrame
The DataFrame that is being compared or tested.
-
expected : DataFrame
The expected result of the operation, for comparison with the actual result.
-
check_row_order : bool, optional
A flag indicating whether the order of rows should be considered in the comparison.
If set to `False` (default), the row order is not taken into account.
If set to `True`, the order of rows is important and will be checked during comparison.
+ rtol : float, optional
+ The relative tolerance, used in asserting approximate equality for float values in df
+ and expected. Set to 1e-5 by default. (See Notes)
+ atol : float, optional
+ The absolute tolerance, used in asserting approximate equality for float values in df
+ and expected. Set to 1e-8 by default. (See Notes)
+
+ Notes
+ -----
+ If the following equation is True for two float values a and b, then they are approximately
+ equal.
+ :math: absolute(a - b) <= (atol + rtol * absolute(b))
Review Comment:
```suggestion
equal. ``absolute(a - b) <= (atol + rtol * absolute(b))``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1262425446
##########
python/pyspark/testing/utils.py:
##########
@@ -250,32 +268,29 @@ def assertDataFrameEqual(df: DataFrame, expected: DataFrame, check_row_order: bo
.config("spark.some.config.option", "some-value").getOrCreate()
>>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # pass
- >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], \
- schema=["id", "amount"])
- >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], \
- schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # fail # doctest: +IGNORE_EXCEPTION_DETAIL
+ >>> assertDataFrameEqual(df1, df2) # DataFrames are equal
+ >>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2, rtol=1e-1) # DataFrames are approx equal by rtol
+ >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)],
+ ... schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)],
+ ... schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
Review Comment:
Can we fix as follows?
```suggestion
>>> assertDataFrameEqual(df1, df2) # DataFrames are equal
explainataion .. blah blah..
>>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2, rtol=1e-1) #
Check equality of float .. blah blah..
>>> df1 = spark.createDataFrame(
... data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(
... data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #41947: [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1263202873
##########
python/pyspark/testing/utils.py:
##########
@@ -336,7 +353,7 @@ def compare_vals(val1, val2):
and all(compare_vals(val1[k], val2[k]) for k in val1.keys())
)
elif isinstance(val1, float) and isinstance(val2, float):
- if abs(val1 - val2) > 1e-5:
+ if abs(val1 - val2) > (atol + rtol * abs(val2)):
Review Comment:
is it possible to directly use the `np.allclose` here?
maybe a few special values to consider: float('nan'), float('inf'), float('-inf')
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org