You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "jeantil (via GitHub)" <gi...@apache.org> on 2024/02/02 15:37:31 UTC
[PR] [JAMES-3978] initial setup of develocity for james build [james-project]
jeantil opened a new pull request, #1972:
URL: https://github.com/apache/james-project/pull/1972
As per https://issues.apache.org/jira/browse/JAMES-3978 this PR configure the basics dor develocity integration.
This brings :
- build scans at https://ge.apache.org
- automatic build scan upload for CI
- manual build scan upload for local dev
- build metrics from the accumulated build scans
- automatic local caching for maven goals that are known by the extension
- can bring speed up gains for both local and ci builds as long as you remember to include the clean goal
https://issues.apache.org/jira/browse/JAMES-3978?focusedCommentId=17813731&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17813731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1476247692
##########
.gitignore:
##########
@@ -6,4 +6,5 @@ dockerfiles/run/**/keystore
dockerfiles/run/**/glowroot
.m2
test-run.log
-build
\ No newline at end of file
+build
+.mvn/.gradle-enterprise/gradle-enterprise-workspace-id
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1476242436
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,54 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
+ <!-- Always publish build scans for CI, only if requested locally -->
+ <publish>#{env['CI'] == null ? 'ON_DEMAND' : 'ALWAYS'}</publish>
+ <capture>
+ <goalInputFiles>false</goalInputFiles> <!-- To be enabled locally when debugging goal caching or if we want to enable predictive test selection -->
Review Comment:
The [documentation](https://docs.gradle.com/enterprise/maven-extension/#capturing_goal_input_files) for this property starts with
> Capturing goal input files increases the amount of data transmitted to the build scan server at the end of the build. If the network connection to the build scan server is poor, it may increase the time required to transmit. Additionally, it may also increase the data storage requirements for the build scan server.
So i intended to enable manually for debugging caching misses but keep it disabled overall at least until we decide to enable/try predictive test selection.
while I'm not too worried about the time to transmit the build scan I was worried about the resources of the apache instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil merged PR #1972:
URL: https://github.com/apache/james-project/pull/1972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "mbaechler (via GitHub)" <gi...@apache.org>.
mbaechler commented on PR #1972:
URL: https://github.com/apache/james-project/pull/1972#issuecomment-1925398961
> We spend 9minutes of cpu time (the elapse is shorter since we do -T1C) fetching git information
Considering we use reactor builds we should be able to fetch this exactly once and reuse the values everywhere instead of repeatedly executing git
Already tried that in the past, i didn't find any solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1477913401
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,54 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
+ <!-- Always publish build scans for CI, only if requested locally -->
+ <publish>#{env['CI'] == null ? 'ON_DEMAND' : 'ALWAYS'}</publish>
+ <capture>
+ <goalInputFiles>false</goalInputFiles> <!-- To be enabled locally when debugging goal caching or if we want to enable predictive test selection -->
Review Comment:
I have seen the entry in apache INFRA and followed your suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on PR #1972:
URL: https://github.com/apache/james-project/pull/1972#issuecomment-1926583437
> > Considering we use reactor builds we should be able to fetch this exactly once and reuse the values everywhere instead of repeatedly executing git
>
> Already tried that in the past, i didn't find any solution.
I was hoping we could use the plugin to generate the properties file in the top level project target then use standard resource processing to copy that file anywhere we actually need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "mbaechler (via GitHub)" <gi...@apache.org>.
mbaechler commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1477100049
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,55 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
+ <!-- Always publish build scans for CI, only if requested locally -->
+ <publish>#{env['CI'] == null ? 'ON_DEMAND' : 'ALWAYS'}</publish>
+ <capture>
+ <goalInputFiles>false</goalInputFiles> <!-- To be enabled locally when debugging goal caching or if we want to enable predictive test selection -->
+ <buildLogging>false</buildLogging> <!-- disabled by default for privacy and performance, to be discussed -->
+ <testLogging>false</testLogging> <!-- disabled by default for privacy and performance, to be discussed -->
+ </capture>
+ <links>
+ <link>
+ <name>Apache James github</name>
+ <url>https://github.com/apache/james-project/</url>
+ </link>
+ <link>
+ <name>Apache James project website</name>
+ <url>https://github.com/apache/james-project/</url>
+ </link>
+ </links>
+ <obfuscation>
+ <!-- Use Spring Expression Language. -->
+ <!-- <hostname>#{isTrue(env['CI']) ? hostname : 'Local agent'}</hostname> -->
+ <hostname>#{isTrue(env['CI']) ? 'jenkins' : 'Local agent'}</hostname>
+ <ipAddresses>#{sha512(ipAddresses)}</ipAddresses>
Review Comment:
it seems quite simple to reverse a sha512 of an ipAddress by computing the 4 billion possible values.
I even expect this table to be already available somewhere, so it's not much of an obfuscation to me.
For precomputed tables, we could at least add a salt.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on PR #1972:
URL: https://github.com/apache/james-project/pull/1972#issuecomment-1928107875
The build seems to consistently fail on these errors despite the rebases
I don't understand how my changes could have affected this
these tests are not even in the `Unstable` test suite ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on PR #1972:
URL: https://github.com/apache/james-project/pull/1972#issuecomment-1925208370
the results of the first [full build](https://ge.apache.org/s/y54epklogxdqu/timeline?cacheability=any-non-cacheable&collapse-all&hide-timeline&outcome=success,failed&toggled=WyI4MTUiXQ&view=by-type) raise interesting questions
- we have javadoc forking maven execution and triggering extra build scans ( I tried to push a fix for that but it didn't work)
- We spend 9minutes of cpu time (the elapse is shorter since we do -T1C fetching git information)
Considering we use reactor builds we should be able to fetch this exactly once and reuse the values everywhere instead of repeatedly executing git
- Scala compilation is not cached and consumes another 9 minutes of compute
- Another 8 minutes consumed in `remote-resources:process` and I have no idea what that is !
the test scan a shows a few more things
- we run checkstyle and scalafix again ( i´ll try to cache thes but the gain is small)
- the top "slow test" projects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1477849953
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,55 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
+ <!-- Always publish build scans for CI, only if requested locally -->
+ <publish>#{env['CI'] == null ? 'ON_DEMAND' : 'ALWAYS'}</publish>
+ <capture>
+ <goalInputFiles>false</goalInputFiles> <!-- To be enabled locally when debugging goal caching or if we want to enable predictive test selection -->
+ <buildLogging>false</buildLogging> <!-- disabled by default for privacy and performance, to be discussed -->
+ <testLogging>false</testLogging> <!-- disabled by default for privacy and performance, to be discussed -->
+ </capture>
+ <links>
+ <link>
+ <name>Apache James github</name>
+ <url>https://github.com/apache/james-project/</url>
+ </link>
+ <link>
+ <name>Apache James project website</name>
+ <url>https://github.com/apache/james-project/</url>
+ </link>
+ </links>
+ <obfuscation>
+ <!-- Use Spring Expression Language. -->
+ <!-- <hostname>#{isTrue(env['CI']) ? hostname : 'Local agent'}</hostname> -->
+ <hostname>#{isTrue(env['CI']) ? 'jenkins' : 'Local agent'}</hostname>
+ <ipAddresses>#{sha512(ipAddresses)}</ipAddresses>
Review Comment:
I didn't think too much about it when copy pasting the sample from the documentation :D
Also I mostly use ipv6 (vpn) which makes the rainbow table much less obvious ;)
more serioulsy I stumbled on the infra wiki recommandations https://cwiki.apache.org/confluence/display/INFRA/Project+Onboarding+Instructions+for+Develocity and applied that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1476235396
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,54 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
Review Comment:
we do it ourselves :D https://github.com/apache/james-project/blob/master/Jenkinsfile#L39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "jeantil (via GitHub)" <gi...@apache.org>.
jeantil commented on PR #1972:
URL: https://github.com/apache/james-project/pull/1972#issuecomment-1924163360
looks like my attempt at preserving the actual CI runner hostname while obfuscating the dev machine names was a failure
```
Caused by: org.codehaus.plexus.component.configurator.ComponentConfigurationException: Cannot evaluate expression '#{isTrue(env['CI']) ? hostname : 'Local agent'}' for configuration entry 'hostname'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] [JAMES-3978] initial setup of develocity for james build [james-project]
Posted by "clayburn (via GitHub)" <gi...@apache.org>.
clayburn commented on code in PR #1972:
URL: https://github.com/apache/james-project/pull/1972#discussion_r1476222550
##########
.gitignore:
##########
@@ -6,4 +6,5 @@ dockerfiles/run/**/keystore
dockerfiles/run/**/glowroot
.m2
test-run.log
-build
\ No newline at end of file
+build
+.mvn/.gradle-enterprise/gradle-enterprise-workspace-id
Review Comment:
I would recommend ignoring all of `.mvn/.gradle-enterprise`. While the only file currently written is the one you are explicitly ignoring, it is possible that later versions of the extension may use this directory.
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,54 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
Review Comment:
From what I see, your CI builds run in Jenkins. From my knowledge, Jenkins does not set a `CI` environment variable. We recommend looking for `JENKINS_URL` instead for these checks in Jenkins.
##########
.mvn/gradle-enterprise.xml:
##########
@@ -0,0 +1,54 @@
+<gradleEnterprise>
+ <server>
+ <url>https://ge.apache.org</url>
+ <allowUntrusted>false</allowUntrusted>
+ <!--
+ credentials provided by jenkins exposing GRADLE_ENTERPRISE_ACCESS_KEY
+ apache members can create their own on the server
+ -->
+ </server>
+ <!-- options are documented at https://docs.gradle.com/enterprise/maven-extension/#configuring_background_uploading -->
+ <buildScan>
+ <backgroundBuildScanUpload>#{isFalse(env['CI'])}</backgroundBuildScanUpload>
+ <!-- Always publish build scans for CI, only if requested locally -->
+ <publish>#{env['CI'] == null ? 'ON_DEMAND' : 'ALWAYS'}</publish>
+ <capture>
+ <goalInputFiles>false</goalInputFiles> <!-- To be enabled locally when debugging goal caching or if we want to enable predictive test selection -->
Review Comment:
I would recommend enabling `goalInputFiles` capture for all builds. This property creates hashes for all file inputs, but does not actually store the files themselves for storage size and privacy reasons. This allows you to troubleshoot builds more effectively, as you may not enable this on a build and then later want this metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org