You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/27 15:56:52 UTC
[GitHub] [beam] mosche opened a new pull request, #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
mosche opened a new pull request, #23388:
URL: https://github.com/apache/beam/pull/23388
Add cron job to trigger Java JMH micro-benchmarks weekly on Sundays (addresses #22238).
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1263081876
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1263136744
# [Codecov](https://codecov.io/gh/apache/beam/pull/23388?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23388](https://codecov.io/gh/apache/beam/pull/23388?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (95d9516) into [master](https://codecov.io/gh/apache/beam/commit/7dd1c6bc4a453e03bc000710d51e5ece65011eb2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7dd1c6b) will **increase** coverage by `0.03%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #23388 +/- ##
==========================================
+ Coverage 73.40% 73.44% +0.03%
==========================================
Files 718 718
Lines 95652 95680 +28
==========================================
+ Hits 70215 70270 +55
+ Misses 24126 24099 -27
Partials 1311 1311
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.21% <0.00%> (+0.04%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/23388?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/metrics/cells.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9jZWxscy5weQ==) | `82.11% <0.00%> (ø)` | |
| [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.07% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/internal/gcp/auth.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvZ2NwL2F1dGgucHk=) | `73.33% <0.00%> (-5.34%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.54% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/metrics/execution.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9leGVjdXRpb24ucHk=) | `87.96% <0.00%> (ø)` | |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `78.48% <0.00%> (ø)` | |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `91.71% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.43% <0.00%> (+0.38%)` | :arrow_up: |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | `76.66% <0.00%> (+1.33%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.26% <0.00%> (+1.69%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/beam/pull/23388/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1260469894
Run Java Jmh benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1260527505
Run Java Jmh benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1325269155
> Do you think we can get it listed on the getting started page http://metrics.beam.apache.org/d/1/getting-started?orgId=1 ?
Absolutely, wasn't aware of the the necessary tag 👍 See https://github.com/apache/beam/pull/24335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1320420380
Do you think we can get it listed on the getting started page http://metrics.beam.apache.org/d/1/getting-started?orgId=1 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1263114298
@apilloud Thanks a lot! The "problem" with JMH benchmarks is that they take a massive amount of time. They are running a large number of iterations including various forks and warmups to produce a more stable and accurate number.
With the current (default) JMH settings these would run for ~ 14 hours:
```
~100 benchmarks * 5 forks * (5 warmup iterations + 5 measurement iterations) * 10 seconds
```
If using just 3 forks it's running for ~ 8.5 hours, which seems more reasonable.
Later, once we have some numbers, we might be able to reduce the number of iterations slightly and see how that impacts (stability of) results.
cc @lukecwik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik merged pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
lukecwik merged PR #23388:
URL: https://github.com/apache/beam/pull/23388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1319612561
Here's the respective dashboard
http://metrics.beam.apache.org/d/kllfR2vVk/java-jmh-benchmarks?orgId=1&from=now-34d&to=now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1308522610
<img width="1724" alt="Screenshot 2022-11-08 at 14 54 26" src="https://user-images.githubusercontent.com/1401430/200802423-b518f361-30d4-4f14-a2ee-a48d87988c20.png">
@lukecwik looking at the benchmark runtimes, we could split the benchmarks into 3 groups and run them 2/3 times weekly
- ByteStringOutputStreamBenchmark: Tue, Sat
- GetterBasedSchemaProviderBenchmark: Thu, Sun
- Everything else: Mon, Wed, Fri
I was briefly looking at implementing this, but without a plugin to conditionally run steps (`conditionalStep`) this would have to be implemented as 3 jobs (or use a shell step to trigger gradle tasks, but loosing all the default switches). Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] apilloud commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
apilloud commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1264137408
Run length of over ~3 hours is going to be disruptive to existing tests regardless of run length. I think the behavior of Jenkins is to just run tests less frequently? Daily would be a good start. Our current tests run every 6 hours and it looks like we are busy ~25% of the time: https://ci-beam.apache.org/label/beam-perf/load-statistics?type=hour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1259767327
Run Java Jmh benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1265765005
We could easily split the job into two JMH runs since there are two JMH projects currently. We can also split it very finely if we utilize `-Pbenchmark=...` and run one suite at a time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1273610311
I agree that `-Pbenchmark=` isn't a great solution because of how much maintenance it would take.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1259711640
Run Java Jmh benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1263084965
Run Java Jmh benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1263174476
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] apilloud commented on pull request #23388: [WIP] Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
apilloud commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1262847867
Our default is to run jobs like this every 6 hours. What are the constraints that limit this to weekly?
(Also you need to run the seed job to get your Jenkins changes to load. I believe `Run Seed Job` worked in the past, but you might need to invoke it via the Jenkins UI.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #23388: Add cron job to trigger Java JMH micro-benchmarks weekly
Posted by GitBox <gi...@apache.org>.
lukecwik commented on PR #23388:
URL: https://github.com/apache/beam/pull/23388#issuecomment-1264111557
It would be great to run this daily or every 2 days. @mosche or @apilloud, do you know how busy or perf machine is already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org