You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/08/28 09:47:43 UTC
[GitHub] [camel-quarkus] ppalaga opened a new issue #1658: Incremental testing on the CI
ppalaga opened a new issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658
I am considering to use GIB https://github.com/vackosar/gitflow-incremental-builder to implement the $subj.
A quick dump of things that are necessary for it to work:
### No "hidden" deps
"Hidden" dependencies of Maven module M are such dependencies, that are required by the module M at build time, but they are not present in the transitive closure of M's dependencies as seen by stock Maven.
The problem occurs e.g. in our integration tests, which explicitly depend only on runtime extension modules, but at build time, they also require their `-deployment` counterparts.
Hidden deps are not a problem for common sequential Maven builds, because the ordering is co-defined by the order of `<module>` elements in the parent-child hierarchy. However, this is an issue for parallel builds using either stock Maven or `mvnd`.
For `mvnd`, we are solving it with `mvnd` specific `<mvnd.builder.rule>`. Clearly, that's not sufficient for GIB, that uses stock maven dependency tree builder. I [learned recently](https://github.com/quarkusio/quarkus/pull/11331#issuecomment-676695992), there is trick how to make hidden dependencies explicit using stock Maven model: one can use a Maven dependency with `<type>pom</type>`, `<scope>test</scope>` that excludes all its transitives via
```
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
```
It is much more XML, but would serve our needs.
Bonu: we will be able to use 'mvn -T1C' both on the CI and locally.
### Check whether GIB handles some corner cases properly:
* Changes of a dependency version in parent (consumed via BoM import) should trigger re-build only of the modules consuming that dependency. GIB will probably not be able to select the optimal set to re-build. If it re-builds all, it is not ideal, but at least, it will not lead to any false positive CI runs.
* Changes of a maven plugin hosted inside the source tree should trigger re-building all modules using the plugin.
Any opnions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684048079
Nice to see more people are considering to use GIB!
> Changes of a dependency version in parent (consumed via BoM import) should trigger re-build only of the modules consuming that dependency.
I am currently working on basic BOM support in GIB: https://github.com/vackosar/gitflow-incremental-builder/issues/241
But I am not sure you mean an even more fine grained approach? E.g. when only a single dependency is updated in the BOM then only those modules that are importing the BOM _and are using this specific dependency_ shall be built?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-800603358
In case you didn't notice already: Quarkus main repo is using GIB for some two weeks now and feedback has been very positive:
E.g.: https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Incremental.20build/near/229472989 (and the following comments)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod edited a comment on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod edited a comment on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684048079
Nice to see that more people are considering to use GIB!
> Changes of a dependency version in parent (consumed via BoM import) should trigger re-build only of the modules consuming that dependency.
I am currently working on basic BOM support in GIB: https://github.com/vackosar/gitflow-incremental-builder/issues/241
But I am not sure you mean an even more fine grained approach? E.g. when only a single dependency is updated in the BOM then only those modules that are importing the BOM _and are using this specific dependency_ shall be built?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684518056
Thanks for the news @famod!
> only a single dependency is updated in the BOM then only those modules that are importing the BOM _and are using this specific dependency_ shall be built?
Yes, that would be ideal. A PR adding a new extension is one of the most typical ones we tend to have. They typically add version properties to the top module and some deps to the BoM. Such changes often do not impact most of the existing modules, so there is no point in building & testing them. My first impression after looking at GIB was that it cannot recognize such fine details and thus would build the whole tree. That's wasting of time, but not utterly wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod edited a comment on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod edited a comment on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684065428
Status: First successful run with Quarkus! A change to `quarkus-bom` triggers 666 (😮) modules, instead of 690 for non-incremental build. 🎉
Nasty catch: `MavenProject.getDependencyManagement()` (for `quarkus-build-parent`) does _not_ contain the `quarkus-bom` dependency anymore! Took me some time to realize that I have to use the depMgmt of the "raw" model instead via `MavenProject.getOriginalModel().getDependencyManagement()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684065428
Status: First successful run with Quarkus! A change to `quarkus-bom` triggers 666 (😮) modules (instead of 690 for non-incremental build). 🎉
Nasty catch: `MavenProject.getDependencyManagement()` (for `quarkus-build-parent`) does _not_ contain the `quarkus-bom` dependency anymore! Took me some time to realize that I have to use the depMgmt of the "raw" model instead via `MavenProject.getOriginalModel().getDependencyManagement()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod edited a comment on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod edited a comment on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684065428
_Edit_: Oops, wrong issue. Sorry!
https://github.com/vackosar/gitflow-incremental-builder/issues/241#issuecomment-684069633
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-689313317
Thanks for the news, @famod . We now have all necessary virtual deps in place (which is also proven by he CI building with `-T1C`). I will continue experimenting with GIB once we have Camel Quarkus 1.1.0 out this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-691706297
@ppalaga FYI, GIB 3.12.0 is out: https://github.com/vackosar/gitflow-incremental-builder/releases/tag/version%2F3.12.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684713299
> My first impression after looking at GIB was that it cannot recognize such fine details and thus would build the whole tree.
Yes, GIB is only looking at which files have changed but not what exactly changed inside of those files.
Parsing a diff to identify the respective dependency is rather complex, see also https://github.com/vackosar/gitflow-incremental-builder/issues/241#issuecomment-684100935 and the subsequent comments.
I am not saying this will never be added to GIB but it will take some time and I guess we cannot catch all cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] gastaldi commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
gastaldi commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-683876681
/cc @famod
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-689168243
@ppalaga In case you want to give 3.12.0-SNAPSHOT a try you can use `3.12.0-20200908.222146-3` from https://oss.sonatype.org/content/repositories/snapshots/com/vackosar/gitflowincrementalbuilder/gitflow-incremental-builder/3.12.0-SNAPSHOT
This includes basic BOM support (and more: https://github.com/vackosar/gitflow-incremental-builder/milestone/17?closed=1).
See also: https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/gitflow-incremental-builder.20%28GIB%29/near/209449771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] famod commented on issue #1658: Incremental testing on the CI
Posted by GitBox <gi...@apache.org>.
famod commented on issue #1658:
URL: https://github.com/apache/camel-quarkus/issues/1658#issuecomment-684718794
> They typically add version properties to the top module and some deps to the BoM.
This case is actually harder to detect than for Quarkus. In Quarkus the version properties are defined in the BOM (= in the same file that also defines the depMgmt) so you do not have to correlate anything between two files (root and BOM).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org