Re: Parallel Build in Camel

[Otavio Rodolfo Piske <an...@gmail.com>]

I was looking at the slides for Peter Palaga, for one of his talks about
Maven/Maven Daemon and it gave me some ideas about trying further
improvements to the build.

I logged them on the following ticket:
https://issues.apache.org/jira/browse/CAMEL-17894

I am not sure when/if I'll have time to look further into this, but I'm
sharing the ticket in case the community has additional ideas/data to
share.

Kind regards

On Wed, Feb 16, 2022 at 6:33 PM Otavio Rodolfo Piske <an...@gmail.com>
wrote:

> Great work, thanks for contributing!
>
> For now I'd keep it as it is, so we can experiment with the parallel
> runner on GH. We have a few more things happening on the Jenkins pipeline
> and it would be better to keep it as stable as possible for now.
>
> Kind regards
>
> On Wed, Feb 16, 2022 at 5:43 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
>> Hi,
>>
>> I just wanted to let you know that with Otavio, we finally could make the
>> Github's runner uses the maven daemon (with 2 threads) as you may have
>> seen. It seems to help a little bit since the validation of a PR now takes
>> between 45-48 minutes while it used to take more than an hour. I know that
>> it is not revolutionary, but I'm afraid that with 2 cores on the runner, it
>> will be hard to do much better.
>>
>> Would it make sense to do the same on the Jenkins' runner?
>>
>> Regards,
>> Nicolas
>>
>> ________________________________
>> From: Nicolas Filotto <nf...@talend.com>
>> Sent: Thursday, February 10, 2022 14:10
>> To: dev@camel.apache.org <de...@camel.apache.org>
>> Subject: Re: Parallel Build in Camel
>>
>> !-------------------------------------------------------------------|
>>   This Message Is From an External Sender
>>   This message came from outside your organization.
>>   Exercise caution when opening attachments or clicking any
>>   links.
>> |-------------------------------------------------------------------!
>>
>> Hi Guillaume,
>>
>> That's awsome, no I didn't know that option, thanks for sharing, it will
>> definitively help.
>>
>> Actually even the maven daemon as it is now (wondeful job by the way)
>> could help.
>> Indeed, on my machine even with 2 threads, the command "mvnd clean
>> install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the
>> equivalent with maven takes 48 minutes.
>>
>> I know that we cannot predict the results on the build pipeline based on
>> this but at least it sounds like it could be interesting to try it, don't
>> you agree?
>>
>> Regards,
>> Nicolas
>>
>> ________________________________
>> From: Guillaume Nodet <gn...@apache.org>
>> Sent: Thursday, February 10, 2022 11:24
>> To: dev@camel.apache.org <de...@camel.apache.org>
>> Subject: Re: Parallel Build in Camel
>>
>> !-------------------------------------------------------------------|
>>   This Message Is From an External Sender
>>   This message came from outside your organization.
>>   Exercise caution when opening attachments or clicking any
>>   links.
>> |-------------------------------------------------------------------!
>>
>> Fwiw, i did a lot of work on the build a while ago, but my main goal was
>> to
>> optimize subsequent  `mvnd -DskipTests` run.
>> Ideall, in such cases, no jars should be changed at all, but that's not
>> the
>> case completely, especially for maven plugins (try running `ls -ltr
>> **/*.jar` after a build). There are pending changes in maven to fix that
>> and I plan to push some changes in camel (mostly plugins updates) once
>> those are released.
>>
>> In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
>> option to gather statistics on the mojo execution time.
>>
>> In addition, I've been working a few weeks ago on the
>> maven-build-cache-extension [1], which is a powerful extension that
>> provides a full per-module build cache.  The cache can be configured
>> locally and remotely (where the output of a module will be downloaded from
>> the remote cache instead of being built locally).  It currently requires
>> some changes in maven-core that will only be released in maven 3.9.0.
>> However, I can try to set up an experimental mvnd branch that would embed
>> this extension if that's of any interest.
>>
>> Guillaume
>>
>> [1]
>> https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$
>>
>> Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <angusyoung@gmail.com
>> >
>> a écrit :
>>
>> > Thanks for looking into this.
>> >
>> > On a related note, I think one area where we could investigate potential
>> > improvements to the compilation is by looking at the performance of our
>> > maven plugins and code generators.
>> >
>> > For example the camel-endpointdsl module takes a long time to generate
>> > sources and build (that one takes almost 2 minutes to run on GH). The
>> > camel-componentdsl one takes about 1 minute. I suspect that if we could
>> > trace why they are so slow, there may be a route for optimizing the
>> build
>> > time (and if we are lucky, that could happen across the whole build).
>> >
>> > A secondary branch of investigation could be looking at whether we can
>> > adjust our build so that we can have it running with "process-classes"
>> > target, to avoid the costly process of compressing and installing the
>> jars.
>> >
>> > Kind regards
>> >
>> > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
>> > wrote:
>> >
>> > > Hi again,
>> > >
>> > > Just to let you know that it appears that there are only 2 cores on
>> the
>> > > target server which is obviously not enough to have an impact on the
>> > build
>> > > time.
>> > >
>> > > Regards,
>> > > Nicolas
>> > > ________________________________
>> > > From: Nicolas Filotto <nf...@talend.com>
>> > > Sent: Wednesday, February 9, 2022 11:33
>> > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > Subject: Re: Parallel Build in Camel
>> > >
>> > > !-------------------------------------------------------------------|
>> > >   This Message Is From an External Sender
>> > >   This message came from outside your organization.
>> > >   Exercise caution when opening attachments or clicking any
>> > >   links.
>> > > |-------------------------------------------------------------------!
>> > >
>> > > Hi,
>> > >
>> > > Thank you very much for your feedbacks. Here is the related PR
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
>> > > .
>> > >
>> > > Regards,
>> > > Nicolas
>> > >
>> > > ________________________________
>> > > From: Otavio Rodolfo Piske <an...@gmail.com>
>> > > Sent: Wednesday, February 9, 2022 10:26
>> > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > Subject: Re: Parallel Build in Camel
>> > >
>> > > !-------------------------------------------------------------------|
>> > >   This Message Is From an External Sender
>> > >   This message came from outside your organization.
>> > >   Exercise caution when opening attachments or clicking any
>> > >   links.
>> > > |-------------------------------------------------------------------!
>> > >
>> > > Hi,
>> > >
>> > > I think it would be OK to give it a try. Please, feel free to open a
>> PR
>> > > with this suggestion.
>> > >
>> > > Kind regards
>> > >
>> > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > If the parallel build is something that can be used in the build
>> > > pipeline,
>> > > > why not simply starting by adding the -T option of maven to the
>> > commands
>> > > > launched by the build (
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
>> > > )
>> > > > as short/middle term solution before eventually considering the
>> maven
>> > > > daemon? What do you think of it?
>> > > >
>> > > > Regards,
>> > > > Nicolas
>> > > >
>> > > > ________________________________
>> > > > From: Otavio Rodolfo Piske <an...@gmail.com>
>> > > > Sent: Wednesday, February 9, 2022 10:03
>> > > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > > Subject: Re: Parallel Build in Camel
>> > > >
>> > > >
>> !-------------------------------------------------------------------|
>> > > >   This Message Is From an External Sender
>> > > >   This message came from outside your organization.
>> > > >   Exercise caution when opening attachments or clicking any
>> > > >   links.
>> > > >
>> |-------------------------------------------------------------------!
>> > > >
>> > > > Hi,
>> > > >
>> > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
>> > like
>> > > > to take a closer look at some of the plugins we have on the build to
>> > see
>> > > if
>> > > > there's some easy speed up we could do.
>> > > >
>> > > > As for the Github, I think we depend on 2 things:
>> > > > 1. I believe we need a Github action that contains and runs maven
>> > daemon
>> > > > 2. I believe this action needs to be approved somehow by the ASF. I
>> am
>> > > not
>> > > > sure about this, though ... maybe others in the community know more
>> > about
>> > > > it.
>> > > >
>> > > > Kind regards
>> > > >
>> > > >
>> > > >
>> > > > As a recipient of an email from Talend, your contact personal data
>> will
>> > > be
>> > > > on our systems. Please see our privacy notice (updated August 2020)
>> at
>> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > Otavio R. Piske
>> > >
>> > >
>> >
>> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
>> > >
>> > > As a recipient of an email from Talend, your contact personal data
>> will
>> > be
>> > > on our systems. Please see our privacy notice (updated August 2020) at
>> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > >
>> > >
>> > >
>> > > As a recipient of an email from Talend, your contact personal data
>> will
>> > be
>> > > on our systems. Please see our privacy notice (updated August 2020) at
>> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > >
>> > >
>> > >
>> >
>> > --
>> > Otavio R. Piske
>> >
>> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$
>> >
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
>> As a recipient of an email from Talend, your contact personal data will
>> be on our systems. Please see our privacy notice (updated August 2020) at
>> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>>
>>
>>
>> As a recipient of an email from Talend, your contact personal data will
>> be on our systems. Please see our privacy notice. <
>> https://www.talend.com/privacy/>
>>
>>
>>
>
> --
> Otavio R. Piske
> http://orpiske.net
>


-- 
Otavio R. Piske
http://orpiske.net