You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/02/23 16:53:00 UTC
[GitHub] [beam] Abacn opened a new pull request, #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Abacn opened a new pull request, #25612:
URL: https://github.com/apache/beam/pull/25612
It is found when reviewing #24015, whenever a new commit pushed, all (65) precommits are triggered at once and it simply causes Jenkins send too much GitHub API, in the following minutes:
- other PR commit push does not trigger jenkins job run in time
- finished jenkins job run stuck at reporting status to GitHub
- phrase not working
Many of these "PreCommit" are splitted from old big JavaPreCommit (for unit test) and JavaPostCommit (for integration tests) #24584. Currently they are triggered automatically when the change is made on java core or buildSrc. I had the concern if this could cause problem when reviewing that PR, apparently it is true. Given that our build system cannot handle these number of PreCommit run at the same time, here I propose disable triggering most of these IO_Direct (21) precommit on buildSrc and core change.
The Con is that when there is a change on dependency version (handled by BeamModulePlugin) one needs to trigger the job by commenting on PR manually now. Considering that every precommit is running every 6 hr and there is another gate on release process (#25609) this should be fine.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442128017
(seed job not triggering, currently jenkins is experience the issues described above, after another push of #24015 done 1 hr ago
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442587942
Thanks @damccorm. Created a "mega" suite only for code change on buildSrc/core/io common; in addition to 21 io precommits on particular IOs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442124597
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442118303
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn merged pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #25612:
URL: https://github.com/apache/beam/pull/25612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442550511
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1447242676
I'm going to merging this as no further change request received. Summary:
For Cron jobs or PR changes only involving specific IO, nothing changed
For PR changes involving
- Default [triggers](https://github.com/apache/beam/blob/62e9a1239bb6f9ef2f1469d0c1af192697f8b10c/.test-infra/jenkins/PrecommitJobBuilder.groovy#L76) (buildSrc, gradle, settings)
- Java core
- Java IO common
- Before: trigger all IO precommits
- After: build task of 21 IO precommits combined into a single precommit task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442116118
R: @kennknowles @damccorm @riteshghorse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442438491
I wonder if a better fix here is to consolidate some of our sharded precommits so that instead of having 1 precommit for IOs (our old state) or 21 precommits for IOs (our current state) we have a precommit per IO if the IO changes frequently and we have 1 precommit (or a few precommits) for the remaining relatively stable ones.
A completely different alternative would be to have individual IO suites that trigger when that IOs directory is modified and 1 "mega IO" suite that triggers when we want to run all of our IO tests. That has the downside of getting out of sync more easily, but does what we want about as efficiently and precisely as possible.
All of this is more of a test structure question for a set of suites I don't personally care a ton about, so I'll defer to you, other IO folks, and @kennknowles (who did the initial sharding, which has been a win IMO), but I don't love the idea of not running precommits that have the potential to break on dependency changes. I also acknowledge, though, that all of these solutions stink because the real solution is to actually fix the infrastructure (which probably means move to GHA for us), but that's not a realistic stopgap solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #25612: Disable triggering most IO_Direct precommit on buildSrc and core change
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25612:
URL: https://github.com/apache/beam/pull/25612#issuecomment-1442561427
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org