You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/09 00:03:35 UTC

[PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

HyukjinKwon opened a new pull request, #57148:
URL: https://github.com/apache/spark/pull/57148

[DO-NOT-MERGE] This is a flaky-test fix under CI verification; the title will be updated with the JIRA id once the fork CI run confirms the fix. Please do not merge yet.

### What changes were proposed in this pull request?

Fix the flaky test `pyspark.sql.tests.test_python_datasource.PythonDataSourceTests.test_data_source_writer_with_logging` on `branch-4.1` by backporting the assertion handling already present on `master` and `branch-4.2` (introduced with SPARK-55799).

The test writes the abort-path DataFrame with `repartitionByRange(2, "age")`, producing two write tasks that each log `"TestJsonWriter.write: abort test"`, and the expected-logs list hardcodes that warning **twice**. When the first partition's task raises, the executor cancels the remaining task, so whether the second `"abort test"` log record is captured before cancellation is timing dependent. The fix splits the assertion:

- `non_abort_logs` (everything except the abort warning) is still asserted exactly, and
- `abort_logs.dropDuplicates(["msg"])` is asserted to contain exactly one row,

so the deterministic logs are checked precisely while the abort warning is checked for presence rather than an exact, race-dependent count.

### Why are the changes needed?

`build_python_3.11.yml` on `branch-4.1` intermittently failed with `[DIFFERENT_ROWS] Results do not match: ( 8.33333 % )` (11 vs 12 rows), the missing row being one of the two identical `"TestJsonWriter.write: abort test"` entries. `master` and `branch-4.2` already handle this; `branch-4.1` was missing the backport.

### Does this PR introduce _any_ user-facing change?

No. Test-only change.

### How was this patch tested?

Existing test `test_data_source_writer_with_logging`. The assertion block is made identical to the one already on `master`/`branch-4.2`; verified via a `build_python_3.11` run on a fork.

### Was this patch authored or co-authored using generative AI tooling?

No.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.

HyukjinKwon commented on PR #57148:
URL: https://github.com/apache/spark/pull/57148#issuecomment-4920494985

   Closing in favor of a direct backport of SPARK-55799 to branch-4.1 (the assertion fix already exists on master and branch-4.2; this is a missing backport rather than a new change).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.

HyukjinKwon closed pull request #57148: [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1
URL: https://github.com/apache/spark/pull/57148


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.

HyukjinKwon commented on PR #57148:
URL: https://github.com/apache/spark/pull/57148#issuecomment-4920508374

   ah it's https://github.com/apache/spark/pull/57100


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.

gaogaotiantian commented on PR #57148:
URL: https://github.com/apache/spark/pull/57148#issuecomment-4920563918

   Yeah I had the PR to backport the test fix. You can merge that I think? I stripped the unrelated change because that's not directly related to the test fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.

HyukjinKwon commented on PR #57148:
URL: https://github.com/apache/spark/pull/57148#issuecomment-4920490289

   yeah I think we did. let me backport


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [DO-NOT-MERGE][PYTHON][TESTS] Fix flaky test_data_source_writer_with_logging on branch-4.1 [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.

HyukjinKwon commented on PR #57148:
URL: https://github.com/apache/spark/pull/57148#issuecomment-4920462993

   I think we fixed this right? @gaogaotiantian 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org