You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/04/24 04:34:00 UTC
[jira] [Resolved] (SPARK-54090) AssertDataframeEqual carries rows when showing differences
[ https://issues.apache.org/jira/browse/SPARK-54090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-54090.
----------------------------------
Fix Version/s: 4.2.0
Resolution: Fixed
Issue resolved by pull request 55522
[https://github.com/apache/spark/pull/55522]
> AssertDataframeEqual carries rows when showing differences
> ----------------------------------------------------------
>
> Key: SPARK-54090
> URL: https://issues.apache.org/jira/browse/SPARK-54090
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.1.0, 4.0.1, 4.2.0
> Reporter: Aimilios Tsouvelekakis
> Assignee: Aimilios Tsouvelekakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> When we try to do assertDataFrameEqual, on two dataframes that have not the same amount of rows, the output gets cascading from the difference till the end. Why this is happening:
> [https://github.com/apache/spark/blob/067969ff946712eeabf47040415f25000837cd87/python/pyspark/testing/utils.py#L1036]
> {code:java}
> def assert_rows_equal(
> rows1: List[Row], rows2: List[Row], maxErrors: int = None, showOnlyDiff: bool = False
> ):
> __tracebackhide__ = True
> zipped = list(zip_longest(rows1, rows2))
> diff_rows_cnt = 0
> diff_rows = []
> has_diff_rows = False
> rows_str1 = ""
> rows_str2 = ""
>
> # count different rows
> for r1, r2 in zipped:
> if not compare_rows(r1, r2):
> diff_rows_cnt += 1
> has_diff_rows = True
> if includeDiffRows:
> diff_rows.append((r1, r2))
> rows_str1 += str(r1) + "\n"
> rows_str2 += str(r2) + "\n"
> if maxErrors is not None and diff_rows_cnt >= maxErrors:
> break
> elif not showOnlyDiff:
> rows_str1 += str(r1) + "\n"
> rows_str2 += str(r2) + "\n"
> generated_diff = _context_diff(
> actual=rows_str1.splitlines(), expected=rows_str2.splitlines(), n=len(zipped)
> )
> if has_diff_rows:
> error_msg = "Results do not match: "
> percent_diff = (diff_rows_cnt / len(zipped)) * 100
> error_msg += "( %.5f %% )" % percent_diff
> error_msg += "\n" + "\n".join(generated_diff)
> data = diff_rows if includeDiffRows else None
> raise PySparkAssertionError(
> errorClass="DIFFERENT_ROWS", messageParameters={"error_msg": error_msg}, data=data
> ){code}
> The problem lies in the way that we zip the lines
> {code:java}
> zipped = list(zip_longest(rows1, rows2)){code}
> With zip longest we assume that the rows are in order and we do position by position comparison but it does not work well with checkRowOrder which defaults to False.
> If I have 1 line difference in 100 line dataframe the result percentage won't be 1% but the amount of rows that cascade towards on from that difference.
> The best solution here would be to have a set based comparison and return the percentage and the rows over that.
> A sample of what zip_longest is doing:
> {code:java}
> from itertools import zip_longest
> rows1 = [ 'A', 'B', 'C', 'D', 'E']
> rows2 = [ 'A', 'C', 'D',]
> zipped = list(zip_longest(rows1, rows2))zipped {code}
> Result:
> {code:java}
> [('A', 'A'), ('B', 'C'), ('C', 'D'), ('D', None), ('E', None)]{code}
> So in this case we would have 80% rows failure.
> This comes directly with the implementation of CheckRowOrder, when it is True we do not sort and it makes sense to use the zip_longest, when it is False it makes sense to use set based comparison since we have already sorted.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org