[PR] fix(tests): make two flaky test suites deterministic [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk opened a new pull request, #1771:
URL: https://github.com/apache/magpie/pull/1771

   Three tests failed intermittently under the full parallel run (the commit hook's `workspace-pytest`, and CI). They blocked commits that had nothing to do with them.
   
   **generate-cve-json: `test_normalised_body.py` saw `DRAFT` where it expected `REVIEW`**
   
   `main(["--config", ...])` reloads the module-level config globals, and nothing restored them. `TestReleaseVoteGating` in `test_cli.py` passes a config with `release_vote_gating = true`. Any later test on the same xdist worker that called `main()` without `--config` ran under that config. Run in that order, the pair failed every time:
   
   ```
   pytest -n0 "tests/test_cli.py::TestReleaseVoteGating::test_gating_on_without_label_stays_draft" \
     tests/test_normalised_body.py::test_a_complete_body_reaches_review
   ```
   
   An autouse fixture in `tests/conftest.py` now puts the config back after any test that changed it. The CLI itself is unchanged: a real process exits after one run, so the leak only exists inside the test process.
   
   **adversarial-review: two runner tests raced the clock**
   
   - `test_one_timeout_does_not_block_the_others` gave codex's stub a shared 5 s budget, already raised once from 1 s.
   - `test_reviewers_run_in_parallel_and_keep_order` required two 3 s sleeps to overlap.
   
   On a loaded machine a fresh stub can start several seconds late. In one failure, one stub started almost 5 s after the other, after the other had already finished its 3 s sleep. So both failed now and then.
   
   Both now use a handshake instead of a time window:
   - In the parallel test, each stub marks that it started and waits up to 20 s for the other's mark.
   - In the timeout test, gemini reports its timeout only after codex has answered. Gemini is listed first, so a sequential run would never start codex.
   
   As a check, I made `run_all` sequential (`max_workers=1`) and confirmed both tests fail. They still test concurrency, not just the absence of a race.
   
   **Verification**
   - The ordered generate-cve-json pair fails on `main` and passes here; the full generate-cve-json suite passed 8 of 8 parallel runs.
   - `tools/adversarial-review/tests/test_runner.py` passed 10 of 10 parallel runs.
   - `prek run --all-files --skip workspace-pytest` is clean. The commit hook's `workspace-pytest` passed.
   
   ---
   
   Generative AI disclosure: this change was prepared with Claude Code (Opus 5.5) and reviewed by me before submission.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(tests): make two flaky test suites deterministic [magpie]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #1771:
URL: https://github.com/apache/magpie/pull/1771


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org