You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "Duhemm (via GitHub)" <gi...@apache.org> on 2024/02/14 13:45:01 UTC
[PR] Cache output of assembly [incubator-pekko]
Duhemm opened a new pull request, #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125
Previously, the build would rebuild over and over the uber jar with the
shaded protobuf3 distribution. With this patch, this is no longer the
case, as the output is cached locally.
------
The issue with assembly re-creating the same uber-jar on (nearly) every run was discovered using the Develocity Timeline feature, which displays task executions on different swimlanes.
Before this fix, the assembly task is clearly shown as being a bottleneck in the build process:
After this patch is applied, the bottleneck is gone:
After the bottleneck is gone, common tasks (eg. `test`) start faster: Gathering all tests’ names goes from 3.7s to 3.2s (-15%).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#issuecomment-1947769555
Thanks for the PR, @Roiocam fyi this is precisely what I was talking about earlier wrt sbt's inbuilt caching functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#issuecomment-1943810571
relates to #1008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#discussion_r1491966858
##########
project/Protobuf.scala:
##########
@@ -38,7 +38,7 @@ object Protobuf {
outputPaths := Seq((Compile / sourceDirectory).value, (Test / sourceDirectory).value).map(_ / "java"),
importPath := None,
// this keeps intellij happy for files that use the shaded protobuf
- Compile / unmanagedJars += (LocalProject("protobuf-v3") / assembly).value,
+ Compile / unmanagedJars += (LocalProject("protobuf-v3") / Compile / packageBin).value,
Review Comment:
This to me also looks suspicious, @Duhemm whats the context behind this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "Duhemm (via GitHub)" <gi...@apache.org>.
Duhemm commented on code in PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#discussion_r1492174440
##########
project/Protobuf.scala:
##########
@@ -38,7 +38,7 @@ object Protobuf {
outputPaths := Seq((Compile / sourceDirectory).value, (Test / sourceDirectory).value).map(_ / "java"),
importPath := None,
// this keeps intellij happy for files that use the shaded protobuf
- Compile / unmanagedJars += (LocalProject("protobuf-v3") / assembly).value,
+ Compile / unmanagedJars += (LocalProject("protobuf-v3") / Compile / packageBin).value,
Review Comment:
`Compile / packagebin` is cached, whereas `assembly` is not. Both tasks produce the same output:
On main:
```
pekko > show protobuf-v3 / Compile / assembly
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
87fdacb73c114cf95d088cb31c164622 -
pekko > show protobuf-v3 / Compile / packageBin
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
87fdacb73c114cf95d088cb31c164622 -
```
With this pull request:
```
pekko > show protobuf-v3 / Compile / assembly
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
aec27c8ab1a2dd13fa5b2f15044089d1 -
pekko > show protobuf-v3 / Compile / packageBin
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
aec27c8ab1a2dd13fa5b2f15044089d1 -
```
So depending on `packageBin` rather than `assembly` should produce the same result without requiring to re-build the jar over and over again.
What's your concern with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "Duhemm (via GitHub)" <gi...@apache.org>.
Duhemm commented on code in PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#discussion_r1492174440
##########
project/Protobuf.scala:
##########
@@ -38,7 +38,7 @@ object Protobuf {
outputPaths := Seq((Compile / sourceDirectory).value, (Test / sourceDirectory).value).map(_ / "java"),
importPath := None,
// this keeps intellij happy for files that use the shaded protobuf
- Compile / unmanagedJars += (LocalProject("protobuf-v3") / assembly).value,
+ Compile / unmanagedJars += (LocalProject("protobuf-v3") / Compile / packageBin).value,
Review Comment:
`Compile / packagebin` is cached, whereas `assembly` is not. Both tasks produce the same output:
On main:
```
pekko > show protobuf-v3 / Compile / assembly
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
87fdacb73c114cf95d088cb31c164622 -
pekko > show protobuf-v3 / Compile / packageBin
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+329-231b9a99-SNAPSHOT.jar
87fdacb73c114cf95d088cb31c164622 -
```
With this pull request:
```
pekko > show protobuf-v3 / Compile / assembly
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
aec27c8ab1a2dd13fa5b2f15044089d1 -
pekko > show protobuf-v3 / Compile / packageBin
(...)
[info] /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
$ md5sum < /Users/martin/Documents/Projects.nosync/apache/incubator-pekko/protobuf-v3/target/scala-2.13/stripped/stripped/pekko-protobuf-v3-assembly-1.1.0-M0+330-f4e48f91-SNAPSHOT.jar
aec27c8ab1a2dd13fa5b2f15044089d1 -
```
So depending on `packageBin` rather than `assembly` should produce the same result without requiring to re-build the jar over and over again.
What's your concern with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Cache output of assembly [incubator-pekko]
Posted by "Roiocam (via GitHub)" <gi...@apache.org>.
Roiocam commented on code in PR #1125:
URL: https://github.com/apache/incubator-pekko/pull/1125#discussion_r1489544363
##########
project/Protobuf.scala:
##########
@@ -38,7 +38,7 @@ object Protobuf {
outputPaths := Seq((Compile / sourceDirectory).value, (Test / sourceDirectory).value).map(_ / "java"),
importPath := None,
// this keeps intellij happy for files that use the shaded protobuf
- Compile / unmanagedJars += (LocalProject("protobuf-v3") / assembly).value,
+ Compile / unmanagedJars += (LocalProject("protobuf-v3") / Compile / packageBin).value,
Review Comment:
This line would be execute osgi bundle unexpected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org