You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2023/08/04 03:48:56 UTC
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42332: [SPARK-44665][PYTHON] Add support for pandas DataFrame assertDataFrameEqual
HyukjinKwon commented on code in PR #42332:
URL: https://github.com/apache/spark/pull/42332#discussion_r1283947475
##########
python/pyspark/testing/pandasutils.py:
##########
@@ -159,13 +160,26 @@ def _assert_pandas_almost_equal(
This function checks if given pandas objects approximately same,
which means the conditions below:
- Both objects are nullable
- - Compare floats rounding to the number of decimal places, 7 after
- dropping missing values (NaN, NaT, None)
+ - Compare decimals and floats according to the following formula:
+ absolute(a - b) <= (atol + rtol * absolute(b))
+ where rtol=1e-5 and atol=1e-8 by numpy convention
"""
# following pandas convention, rtol=1e-5 and atol=1e-8
rtol = 1e-5
atol = 1e-8
+ def compare_vals_approx(val1, val2):
+ # compare vals for approximate equality
+ if isinstance(lval, (float, decimal.Decimal)) or isinstance(rval, (float, decimal.Decimal)):
+ print("\nentered if float or decimal\n")
Review Comment:
can we remove print
##########
python/pyspark/testing/pandasutils.py:
##########
@@ -159,13 +160,26 @@ def _assert_pandas_almost_equal(
This function checks if given pandas objects approximately same,
which means the conditions below:
- Both objects are nullable
- - Compare floats rounding to the number of decimal places, 7 after
- dropping missing values (NaN, NaT, None)
+ - Compare decimals and floats according to the following formula:
+ absolute(a - b) <= (atol + rtol * absolute(b))
+ where rtol=1e-5 and atol=1e-8 by numpy convention
"""
# following pandas convention, rtol=1e-5 and atol=1e-8
rtol = 1e-5
atol = 1e-8
+ def compare_vals_approx(val1, val2):
+ # compare vals for approximate equality
+ if isinstance(lval, (float, decimal.Decimal)) or isinstance(rval, (float, decimal.Decimal)):
+ print("\nentered if float or decimal\n")
Review Comment:
can we remove print
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org