You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mikebridge (via GitHub)" <gi...@apache.org> on 2026/10/20 23:47:37 UTC

[PR] ci(python): try worksteal for unit tests [superset]

mikebridge opened a new pull request, #45549:
URL: https://github.com/apache/superset/pull/45549

   ### SUMMARY
   The Python unit job uses pytest-xdist's `loadfile` scheduling, which keeps a worker assigned to one file even when other workers become idle. Try `worksteal` so idle workers can take pending tests from busy workers. This is a separate experiment from the coverage-gate change, allowing the CI timing and stability of each to be measured independently.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   Not applicable; CI workflow only.
   
   ### TESTING INSTRUCTIONS
   - Compare the `Python unit tests` step duration with recent runs of the same job using `loadfile`.
   - Re-run the job to look for worker-distribution flakes before deciding whether to keep this change.
   - Locally, `pytest -n 4 --dist worksteal` passed 4,679 tests and skipped 3 across the SQL and semantic-layer suites and two files with module-scoped fixtures. The workflow's YAML and security hooks passed.
   
   ### ADDITIONAL INFORMATION
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] ci(python): try worksteal for unit tests [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge closed pull request #45549: ci(python): try worksteal for unit tests
URL: https://github.com/apache/superset/pull/45549


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] ci(python): try worksteal for unit tests [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #45549:
URL: https://github.com/apache/superset/pull/45549#issuecomment-6304670866

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/45549?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 81.07%. Comparing base ([`4e7e2b7`](https://app.codecov.io/gh/apache/superset/commit/4e7e2b7390d3600a8808d29d5f7595eca1cc9c6c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9b0b9f3`](https://app.codecov.io/gh/apache/superset/commit/9b0b9f38351bc4dee3a62abe6fc2be4fb3307f18?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #45549      +/-   ##
   ==========================================
   - Coverage   83.98%   81.07%   -2.92%     
   ==========================================
     Files        3055     3055              
     Lines      201017   201017              
     Branches    46949    46949              
   ==========================================
   - Hits       168828   162966    -5862     
   - Misses      28943    34613    +5670     
   - Partials     3246     3438     +192     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.89% <ø> (ø)` | |
   | [mysql](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [postgres](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.65% <ø> (ø)` | |
   | [python](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.90% <ø> (-5.30%)` | :arrow_down: |
   | [sqlite](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/45549/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.40% <ø> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/45549?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] ci(python): try worksteal for unit tests [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #45549:
URL: https://github.com/apache/superset/pull/45549#issuecomment-6351713244

   Closing: two confirmation runs (28:50 and 18:51 for the main step) were within master's normal 19–30 minute range, so worksteal scheduling showed no repeatable gain over loadfile. The coverage-reuse change in #45548 is the one worth keeping.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org