You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "samotleriche (via GitHub)" <gi...@apache.org> on 2023/12/26 19:24:15 UTC

[PR] Samotleriche/optimizing cache build (eventmesh)

samotleriche opened a new pull request, #4689:
URL: https://github.com/apache/eventmesh/pull/4689

   ### Motivation
   
   The motivation of this PR is to improve the build time by resolving caching issues.
   
   - build cache was not enabled.
   - spotlessApply runs on every build, modifying source code and causing cache misses.
   - spotless target included build directory, causing cache misses due to changing inputs.
   - WatchFileManagerTest edits a properties file in place causing a cache miss in subsequent tasks.
   
   ### Modifications
   
   - Enabled gradle build cache by setting org.gradle.caching = true in the gradle.properties file.
   - Removed checkstyleMain dependency on spotlessApply (should run spotlessApply as a separate task or integrate into CI).
   - Limited the target of spotlessJava to only the src java files.
   - Copied configuration.properties into a TempDir for testing the onChange method.
   
   These changes improved the build time for back-to-back builds with no changes from ~4 minutes 30 seconds down to ~15 seconds. 
   
   build scan before modifications: https://ge.solutions-team.gradle.com/s/wl6q5fikzjhnq#performance
   build scan after modifications: https://ge.solutions-team.gradle.com/s/cssmzbjx5v7js#performance
   
   ### Documentation
   
   - Does this pull request introduce a new feature? no
   - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) n/a
   - If a feature is not applicable for documentation, explain why? n/a
   - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1874924365

   @samotleriche 
   
   Hi! The WatchFileManagerTest unit test failed. May you please help have a check? https://github.com/apache/eventmesh/issues/4711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] Build caching optimizations (eventmesh)

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1870761851

   ## [Codecov](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`1a381ed`)](https://app.codecov.io/gh/apache/eventmesh/commit/1a381ed2d54f4b2fbe0ffdf4d9d0ae59ebcc0e30?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.39% compared to head [(`e76ff8d`)](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.38%.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #4689      +/-   ##
   ============================================
   - Coverage     17.39%   17.38%   -0.02%     
     Complexity     1759     1759              
   ============================================
     Files           797      797              
     Lines         29850    29845       -5     
     Branches       2579     2577       -2     
   ============================================
   - Hits           5192     5188       -4     
   + Misses        24177    24174       -3     
   - Partials        481      483       +2     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/eventmesh/pull/4689?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1872226037

   > > able to pull the results of the test task from cache
   > 
   > So, when a class with unit test is modified, the unit test for that class will be automatically executed again, right?
   > 
   > > The default value of `removeUnusedEntriesAfterDays` is 7
   > 
   > I'm referring to that parameter as well. Since it has a default value, there's no need to set it separately.
   
   Yes, thats correct. Any test tasks that have that class as an input will be re-executed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871459239

   @samotleriche It would be great to associate this PR with the issue https://github.com/apache/eventmesh/issues/1054 you have found.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 merged PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "yanrongzhen (via GitHub)" <gi...@apache.org>.
yanrongzhen commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1872228438

   @samotleriche I think maybe we shouldn't change the behavior of modifying source code through spotless during build. I don't know how spotless will affect this pr if it is not modified.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1873616991

   The changes made after running spotlessJava are not committed to the master branch; instead, they are discarded. We can run spotlessJava separately before each release and commit the formatted changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] Build caching optimizations (eventmesh)

Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1869949973

   @yanrongzhen Could you please help review the impact of the modification of this PR on the use of Spotless?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1885307700

   @samotleriche @clayburn @tylerbertrand 
   
   Hi, there is a CI failure issue likely caused by enabling Gradle build caching. https://github.com/apache/eventmesh/issues/4737
   
   May you please help have a check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1869859615

   It seems the Build Scan with cache didn't execute tasks: https://ge.solutions-team.gradle.com/s/cssmzbjx5v7js#tests. Can we run tests on top of using a build cache?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871618860

   @Pil0tXia the issue has been linked to the PR 🙂
   
   Regarding your two questions:
   
   1. This optimization allows subsequent builds to pull the outputs of a task from cache, rather than re-executing the task, if the task's inputs have not changed. The first build runs all the tests and the second build (the one linked which shows no tasks executed) is able to pull the results of the test task from cache.
   
   2 . The local build cache is a DirectoryBuildCache, which by default removes unused cache entries after 7 days ([ref](https://docs.gradle.org/current/dsl/org.gradle.caching.local.DirectoryBuildCache.html#org.gradle.caching.local.Dire[%E2%80%A6]e:removeUnusedEntriesAfterDays)), but can be configured via the removeUnusedEntriesAfterDays property. Typically the default is fine, but if you need to configure this, it is recommended to configure this via an init script, for the reasons described [here](https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure_local).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "tylerbertrand (via GitHub)" <gi...@apache.org>.
tylerbertrand commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1873549078

   Hi @yanrongzhen. Jumping in for @samotleriche here - we definitely understand if you don't want to change the current behavior of executing `spotlessApply` during a build. It is indeed a bit of a process change. Our other optimizations - enabling caching, and limiting spotless to just the src java files - will still have a positive impact on build times.
   
   To quantify the impact of continuing to run `spotlessApply` on every `build`... With the improperly formatted code currently committed to `master`, [here is a build scan](https://ge.solutions-team.gradle.com/s/uno2ij2h4csgw/timeline?collapse-all&outcome=success&view=by-type) showing all the tasks that re-run needlessly if we keep the `checkstyleMain` dependency on `spotlessApply`. This will happen to some degree any time a developer makes a code change that is not properly formatted and runs a `build` - `spotlessApply` will format the code, and on the subsequent build, compilation, tests, etc. will all re-run even if no further changes were made by the developer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "tylerbertrand (via GitHub)" <gi...@apache.org>.
tylerbertrand commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1875542347

   Hey @Pil0tXia, I'll take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #1054] Build caching optimizations (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1874849826

   @samotleriche Hi! The `WatchFileManagerTest` failed during build. May you please help have a check?
   
   https://github.com/apache/eventmesh/issues/4711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] Build caching optimizations (eventmesh)

Posted by "samotleriche (via GitHub)" <gi...@apache.org>.
samotleriche commented on PR #4689:
URL: https://github.com/apache/eventmesh/pull/4689#issuecomment-1871387354

   @Pil0tXia should I link this PR to this issue that was created back in 2022 or create a new issue?
   https://github.com/apache/eventmesh/issues/1054
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org