You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/23 20:33:57 UTC

[PR] [SPARK-54090][PYTHON] Fix cascading diff in assertDataFrameEqual when row counts differ [spark]

Yicong-Huang opened a new pull request, #55522:
URL: https://github.com/apache/spark/pull/55522

   ### What changes were proposed in this pull request?
   
   When `assertDataFrameEqual` is called with `checkRowOrder=False` (the default) and the two inputs have **different row counts**, a single missing/extra row cascades into a mismatch on every subsequent row. This inflates both the diff count and the reported mismatch percentage.
   
   Root cause: after sorting both lists by `str(row)`, `assert_rows_equal` pairs rows with `zip_longest`. When one side is shorter, the pairing shifts past every row following the hole.
   
   Fix: switch to a merge-walk over the sorted lists **only when their lengths differ**. Equal lengths keep `zip_longest` so that field-level diffs continue to be reported as paired rows (preserving existing docstrings and tests that rely on "B vs X" style pairing).
   
   The merge-walk uses `compare_rows` for equality (honoring `rtol`/`atol`) and `str(r)` for ordering decisions (consistent with how the lists were sorted).
   
   ### Why are the changes needed?
   
   Reproducer:
   
   ```python
   from pyspark.testing.utils import assertDataFrameEqual
   from pyspark.sql import Row
   
   actual   = [Row(id='1'), Row(id='2'), Row(id='3'), Row(id='4'), Row(id='5')]
   expected = [Row(id='1'), Row(id='2'),              Row(id='4'), Row(id='5')]
   assertDataFrameEqual(actual, expected)
   ```
   
   Before this fix: `Results do not match: ( 60.00000 % )` (3 of 5 rows reported as different).
   After this fix:  `Results do not match: ( 20.00000 % )` (only `Row(id='3')` is reported).
   
   A larger example from the JIRA: rows1 has 5 rows, rows2 has 3 of them missing in the middle -- the old code reports 80% mismatch; the new code reports 40%, matching what a user would expect from a sorted-set comparison.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, but only in the error message / reported data when `assertDataFrameEqual(checkRowOrder=False)` is given inputs whose row counts differ:
   - The reported mismatch percentage is no longer inflated by positional shifting.
   - `includeDiffRows=True` now returns `(row, None)` tuples for extras and `(None, row)` tuples for missing rows, rather than shifted `(row, row)` pairs.
   
   Behavior when row counts are equal (including all existing docstring examples) is unchanged.
   
   ### How was this patch tested?
   
   - Added five targeted tests in `python/pyspark/sql/tests/test_utils.py`:
     - `test_different_row_count_middle_missing_no_cascading_diff`
     - `test_different_row_count_multiple_missing`
     - `test_different_row_count_includeDiffRows`
     - `test_different_row_count_mixed_extra_and_missing`
     - `test_different_row_count_extras_at_end`
   - Ran the full `test_utils.py` suite: 81 passed.
   - Verified the docstring examples (`maxErrors`, `showOnlyDiff`, `includeDiffRows`) still produce their documented output.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54090][PYTHON] Fix cascading diff in assertDataFrameEqual when row counts differ [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #55522: [SPARK-54090][PYTHON] Fix cascading diff in assertDataFrameEqual when row counts differ
URL: https://github.com/apache/spark/pull/55522


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54090][PYTHON] Fix cascading diff in assertDataFrameEqual when row counts differ [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #55522:
URL: https://github.com/apache/spark/pull/55522#issuecomment-4310592687

   cc @HyukjinKwon @zhengruifeng 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54090][PYTHON] Fix cascading diff in assertDataFrameEqual when row counts differ [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #55522:
URL: https://github.com/apache/spark/pull/55522#issuecomment-4310608107

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org