You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "samotleriche (via GitHub)" <gi...@apache.org> on 2023/12/26 19:24:15 UTC
[PR] Samotleriche/optimizing cache build (eventmesh)
samotleriche opened a new pull request, #4689:
URL: https://github.com/apache/eventmesh/pull/4689
### Motivation
The motivation of this PR is to improve the build time by resolving caching issues.
- build cache was not enabled.
- spotlessApply runs on every build, modifying source code and causing cache misses.
- spotless target included build directory, causing cache misses due to changing inputs.
- WatchFileManagerTest edits a properties file in place causing a cache miss in subsequent tasks.
### Modifications
- Enabled gradle build cache by setting org.gradle.caching = true in the gradle.properties file.
- Removed checkstyleMain dependency on spotlessApply (should run spotlessApply as a separate task or integrate into CI).
- Limited the target of spotlessJava to only the src java files.
- Copied configuration.properties into a TempDir for testing the onChange method.
These changes improved the build time for back-to-back builds with no changes from ~4 minutes 30 seconds down to ~15 seconds.
build scan before modifications: https://ge.solutions-team.gradle.com/s/wl6q5fikzjhnq#performance
build scan after modifications: https://ge.solutions-team.gradle.com/s/cssmzbjx5v7js#performance
### Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) n/a
- If a feature is not applicable for documentation, explain why? n/a
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1874924365
@samotleriche
Hi! The WatchFileManagerTest unit test failed. May you please help have a check? https://github.com/apache/eventmesh/issues/4711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] Build caching optimizations (eventmesh)
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1870761851
## [Codecov](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`1a381ed`)](https://app.codecov.io/gh/apache/eventmesh/commit/1a381ed2d54f4b2fbe0ffdf4d9d0ae59ebcc0e30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.39% compared to head [(`e76ff8d`)](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.38%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4689 +/- ##
============================================
- Coverage 17.39% 17.38% -0.02%
Complexity 1759 1759
============================================
Files 797 797
Lines 29850 29845 -5
Branches 2579 2577 -2
============================================
- Hits 5192 5188 -4
+ Misses 24177 24174 -3
- Partials 481 483 +2
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1872226037
> > able to pull the results of the test task from cache
>
> So, when a class with unit test is modified, the unit test for that class will be automatically executed again, right?
>
> > The default value of `removeUnusedEntriesAfterDays` is 7
>
> I'm referring to that parameter as well. Since it has a default value, there's no need to set it separately.
Yes, thats correct. Any test tasks that have that class as an input will be re-executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871459239
@samotleriche It would be great to associate this PR with the issue https://github.com/apache/eventmesh/issues/1054 you have found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 merged PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "yanrongzhen (via GitHub)" <gi...@apache.org>.
yanrongzhen commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1872228438
@samotleriche I think maybe we shouldn't change the behavior of modifying source code through spotless during build. I don't know how spotless will affect this pr if it is not modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1873616991
The changes made after running spotlessJava are not committed to the master branch; instead, they are discarded. We can run spotlessJava separately before each release and commit the formatted changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] Build caching optimizations (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1869949973
@yanrongzhen Could you please help review the impact of the modification of this PR on the use of Spotless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1885307700
@samotleriche @clayburn @tylerbertrand
Hi, there is a CI failure issue likely caused by enabling Gradle build caching. https://github.com/apache/eventmesh/issues/4737
May you please help have a check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1869859615
It seems the Build Scan with cache didn't execute tasks: https://ge.solutions-team.gradle.com/s/cssmzbjx5v7js#tests. Can we run tests on top of using a build cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871618860
@Pil0tXia the issue has been linked to the PR 🙂
Regarding your two questions:
1. This optimization allows subsequent builds to pull the outputs of a task from cache, rather than re-executing the task, if the task's inputs have not changed. The first build runs all the tests and the second build (the one linked which shows no tasks executed) is able to pull the results of the test task from cache.
2 . The local build cache is a DirectoryBuildCache, which by default removes unused cache entries after 7 days ([ref](https://docs.gradle.org/current/dsl/org.gradle.caching.local.DirectoryBuildCache.html#org.gradle.caching.local.Dire[%E2%80%A6]e:removeUnusedEntriesAfterDays)), but can be configured via the removeUnusedEntriesAfterDays property. Typically the default is fine, but if you need to configure this, it is recommended to configure this via an init script, for the reasons described [here](https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure_local).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "tylerbertrand (via GitHub)" <gi...@apache.org>.
tylerbertrand commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1873549078
Hi @yanrongzhen. Jumping in for @samotleriche here - we definitely understand if you don't want to change the current behavior of executing `spotlessApply` during a build. It is indeed a bit of a process change. Our other optimizations - enabling caching, and limiting spotless to just the src java files - will still have a positive impact on build times.
To quantify the impact of continuing to run `spotlessApply` on every `build`... With the improperly formatted code currently committed to `master`, [here is a build scan](https://ge.solutions-team.gradle.com/s/uno2ij2h4csgw/timeline?collapse-all&outcome=success&view=by-type) showing all the tasks that re-run needlessly if we keep the `checkstyleMain` dependency on `spotlessApply`. This will happen to some degree any time a developer makes a code change that is not properly formatted and runs a `build` - `spotlessApply` will format the code, and on the subsequent build, compilation, tests, etc. will all re-run even if no further changes were made by the developer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "tylerbertrand (via GitHub)" <gi...@apache.org>.
tylerbertrand commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1875542347
Hey @Pil0tXia, I'll take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1874849826
@samotleriche Hi! The `WatchFileManagerTest` failed during build. May you please help have a check?
https://github.com/apache/eventmesh/issues/4711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] Build caching optimizations (eventmesh)
Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871387354
@Pil0tXia should I link this PR to this issue that was created back in 2022 or create a new issue?
https://github.com/apache/eventmesh/issues/1054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org