You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "asl3 (via GitHub)" <gi...@apache.org> on 2023/07/28 02:30:04 UTC
[GitHub] [spark] asl3 opened a new pull request, #42196: Improve assertDataFrameEqual error message format
asl3 opened a new pull request, #42196:
URL: https://github.com/apache/spark/pull/42196
### What changes were proposed in this pull request?
This PR improves the error message format for `assertDataFrameEqual` by printing the entire DataFrame.
### Why are the changes needed?
The change is needed to clarify the error message for unequal DataFrames.
### Does this PR introduce _any_ user-facing change?
Yes, the PR affects the error message display for users. See the new error message output examples below.
### How was this patch tested?
Added tests to `python/pyspark/sql/tests/test_utils.py` and `python/pyspark/sql/tests/connect/test_utils.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #42196: [SPARK-44218] Customize diff log in assertDataFrameEqual error message format
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #42196:
URL: https://github.com/apache/spark/pull/42196#discussion_r1279694241
##########
python/pyspark/sql/tests/test_utils.py:
##########
@@ -287,25 +302,30 @@ def test_assert_notequal_arraytype(self):
),
)
- expected_error_message = "Results do not match: "
- percent_diff = (1 / 2) * 100
- expected_error_message += "( %.5f %% )" % percent_diff
+ if isinstance(df2, DataFrame):
+ actual_str = df1._jdf.showString(2, 2, False)
+ expected_str = df2._jdf.showString(2, 2, False)
+ else:
+ # Spark Connect
+ actual_str = df1._show_string(2, 2, False)
+ expected_str = df2._show_string(2, 2, False)
Review Comment:
Instead of using _show_string to invoke a Spark job, how about we convert the df to a pandas dataframe. Pandas should be a required dependency when spark connect is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42196: [SPARK-44218][PYTHON] Customize diff log in assertDataFrameEqual error message format
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42196:
URL: https://github.com/apache/spark/pull/42196#issuecomment-1661247588
Merged to master and branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42196: [SPARK-44218][PYTHON] Customize diff log in assertDataFrameEqual error message format
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42196: [SPARK-44218][PYTHON] Customize diff log in assertDataFrameEqual error message format
URL: https://github.com/apache/spark/pull/42196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42196: Improve assertDataFrameEqual error message format
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42196:
URL: https://github.com/apache/spark/pull/42196#discussion_r1277112278
##########
python/pyspark/testing/utils.py:
##########
@@ -189,6 +190,39 @@ def search_jar(project_relative_path, sbt_jar_name_prefix, mvn_jar_name_prefix):
return jars[0]
+def _context_diff(actual: List[str], expected: List[str], n: int = 3):
+ """
+ Modified from difflib context_diff API,
+ see original code here: https://github.com/python/cpython/blob/main/Lib/difflib.py#L1180
+ """
+ def red(s: str) -> str:
+ RedColor = "\033[31m"
+ NoColor = "\033[0m"
+ return RedColor + str(s) + NoColor
Review Comment:
```suggestion
red_color = "\033[31m"
no_color = "\033[0m"
return red_color + str(s) + no_color
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42196: Improve assertDataFrameEqual error message format
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42196:
URL: https://github.com/apache/spark/pull/42196#discussion_r1277112777
##########
python/pyspark/testing/utils.py:
##########
@@ -189,6 +190,39 @@ def search_jar(project_relative_path, sbt_jar_name_prefix, mvn_jar_name_prefix):
return jars[0]
+def _context_diff(actual: List[str], expected: List[str], n: int = 3):
+ """
+ Modified from difflib context_diff API,
+ see original code here: https://github.com/python/cpython/blob/main/Lib/difflib.py#L1180
+ """
+ def red(s: str) -> str:
Review Comment:
I think we'll need a configuration to disable the colors. In some terminals, they do not work, and looks weird characters. One reference is `sbt -no-colors`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42196: Improve assertDataFrameEqual error message format
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42196:
URL: https://github.com/apache/spark/pull/42196#issuecomment-1655060579
Otherwise, I think it's good to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org