You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Nicolas Filotto <nf...@talend.com> on 2022/02/08 16:27:30 UTC
Parallel Build in Camel
Hi Cameleers,
I was wondering if it was safe to use the parallel build of maven at least to compile camel faster. Are you aware of any issues with it?
I'm asking that because on my local machine (MBP 2,6 GHz Intel Core i7 with 6 cores), launching:
* mvn clean install -Pfastinstall -Psourcecheck - takes about 50 minutes
* mvn -T 1C clean install -Pfastinstall -Psourcecheck - takes about 30 minutes
So, it is not revolutionary, but it can still be helpful if it is reliable/safe enough.
Thanks in advance for your answers,
Nicolas
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by JB Onofré <jb...@nanthrax.net>.
Hi Nicolas
Guillaume worked on maven daemon feature (with parallel build) specifically for Camel at the beginning.
So yeah you can use it.
Personally I use it sometime when I do a full build (which is rare as I build module per module most of the time).
Regards
JB
> Le 8 févr. 2022 à 17:35, Otavio Rodolfo Piske <an...@gmail.com> a écrit :
>
> Hello,
>
> I have been using parallel builds with Camel and all it's sub-projects
> regularly. On some (rare) occasions I find a problem or two, but it's
> usually solved with rebuilding or running a non-parallel build.
>
> In my case, I build using Maven Daemon [1] and it has proven to be very
> stable in terms of reliability. A cached build of Camel takes about 1
> minute on my Desktop (Ryzen 9 3900XT 12 cores / 24 threads, 32Gb RAM). I
> don't have the precise numbers, but a clean build with Maven Daemon takes
> about 9 minutes. On my MBP with a similar configuration to yours, a cached
> build is around 3-4 minutes and a non-cached is between 14 to 18 minutes.
> Not great, but not terrible either.
>
> 1. https://github.com/apache/maven-mvnd
>
> Kind regards
>
>> On Tue, Feb 8, 2022 at 5:27 PM Nicolas Filotto <nf...@talend.com> wrote:
>>
>> Hi Cameleers,
>>
>> I was wondering if it was safe to use the parallel build of maven at least
>> to compile camel faster. Are you aware of any issues with it?
>>
>> I'm asking that because on my local machine (MBP 2,6 GHz Intel Core i7
>> with 6 cores), launching:
>>
>> * mvn clean install -Pfastinstall -Psourcecheck - takes about 50
>> minutes
>> * mvn -T 1C clean install -Pfastinstall -Psourcecheck - takes about 30
>> minutes
>>
>> So, it is not revolutionary, but it can still be helpful if it is
>> reliable/safe enough.
>>
>> Thanks in advance for your answers,
>> Nicolas
>>
>> As a recipient of an email from Talend, your contact personal data will be
>> on our systems. Please see our privacy notice (updated August 2020) at
>> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>>
>>
>>
>
> --
> Otavio R. Piske
> http://orpiske.net
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
I was looking at the slides for Peter Palaga, for one of his talks about
Maven/Maven Daemon and it gave me some ideas about trying further
improvements to the build.
I logged them on the following ticket:
https://issues.apache.org/jira/browse/CAMEL-17894
I am not sure when/if I'll have time to look further into this, but I'm
sharing the ticket in case the community has additional ideas/data to
share.
Kind regards
On Wed, Feb 16, 2022 at 6:33 PM Otavio Rodolfo Piske <an...@gmail.com>
wrote:
> Great work, thanks for contributing!
>
> For now I'd keep it as it is, so we can experiment with the parallel
> runner on GH. We have a few more things happening on the Jenkins pipeline
> and it would be better to keep it as stable as possible for now.
>
> Kind regards
>
> On Wed, Feb 16, 2022 at 5:43 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
>> Hi,
>>
>> I just wanted to let you know that with Otavio, we finally could make the
>> Github's runner uses the maven daemon (with 2 threads) as you may have
>> seen. It seems to help a little bit since the validation of a PR now takes
>> between 45-48 minutes while it used to take more than an hour. I know that
>> it is not revolutionary, but I'm afraid that with 2 cores on the runner, it
>> will be hard to do much better.
>>
>> Would it make sense to do the same on the Jenkins' runner?
>>
>> Regards,
>> Nicolas
>>
>> ________________________________
>> From: Nicolas Filotto <nf...@talend.com>
>> Sent: Thursday, February 10, 2022 14:10
>> To: dev@camel.apache.org <de...@camel.apache.org>
>> Subject: Re: Parallel Build in Camel
>>
>> !-------------------------------------------------------------------|
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Exercise caution when opening attachments or clicking any
>> links.
>> |-------------------------------------------------------------------!
>>
>> Hi Guillaume,
>>
>> That's awsome, no I didn't know that option, thanks for sharing, it will
>> definitively help.
>>
>> Actually even the maven daemon as it is now (wondeful job by the way)
>> could help.
>> Indeed, on my machine even with 2 threads, the command "mvnd clean
>> install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the
>> equivalent with maven takes 48 minutes.
>>
>> I know that we cannot predict the results on the build pipeline based on
>> this but at least it sounds like it could be interesting to try it, don't
>> you agree?
>>
>> Regards,
>> Nicolas
>>
>> ________________________________
>> From: Guillaume Nodet <gn...@apache.org>
>> Sent: Thursday, February 10, 2022 11:24
>> To: dev@camel.apache.org <de...@camel.apache.org>
>> Subject: Re: Parallel Build in Camel
>>
>> !-------------------------------------------------------------------|
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Exercise caution when opening attachments or clicking any
>> links.
>> |-------------------------------------------------------------------!
>>
>> Fwiw, i did a lot of work on the build a while ago, but my main goal was
>> to
>> optimize subsequent `mvnd -DskipTests` run.
>> Ideall, in such cases, no jars should be changed at all, but that's not
>> the
>> case completely, especially for maven plugins (try running `ls -ltr
>> **/*.jar` after a build). There are pending changes in maven to fix that
>> and I plan to push some changes in camel (mostly plugins updates) once
>> those are released.
>>
>> In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
>> option to gather statistics on the mojo execution time.
>>
>> In addition, I've been working a few weeks ago on the
>> maven-build-cache-extension [1], which is a powerful extension that
>> provides a full per-module build cache. The cache can be configured
>> locally and remotely (where the output of a module will be downloaded from
>> the remote cache instead of being built locally). It currently requires
>> some changes in maven-core that will only be released in maven 3.9.0.
>> However, I can try to set up an experimental mvnd branch that would embed
>> this extension if that's of any interest.
>>
>> Guillaume
>>
>> [1]
>> https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$
>>
>> Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <angusyoung@gmail.com
>> >
>> a écrit :
>>
>> > Thanks for looking into this.
>> >
>> > On a related note, I think one area where we could investigate potential
>> > improvements to the compilation is by looking at the performance of our
>> > maven plugins and code generators.
>> >
>> > For example the camel-endpointdsl module takes a long time to generate
>> > sources and build (that one takes almost 2 minutes to run on GH). The
>> > camel-componentdsl one takes about 1 minute. I suspect that if we could
>> > trace why they are so slow, there may be a route for optimizing the
>> build
>> > time (and if we are lucky, that could happen across the whole build).
>> >
>> > A secondary branch of investigation could be looking at whether we can
>> > adjust our build so that we can have it running with "process-classes"
>> > target, to avoid the costly process of compressing and installing the
>> jars.
>> >
>> > Kind regards
>> >
>> > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
>> > wrote:
>> >
>> > > Hi again,
>> > >
>> > > Just to let you know that it appears that there are only 2 cores on
>> the
>> > > target server which is obviously not enough to have an impact on the
>> > build
>> > > time.
>> > >
>> > > Regards,
>> > > Nicolas
>> > > ________________________________
>> > > From: Nicolas Filotto <nf...@talend.com>
>> > > Sent: Wednesday, February 9, 2022 11:33
>> > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > Subject: Re: Parallel Build in Camel
>> > >
>> > > !-------------------------------------------------------------------|
>> > > This Message Is From an External Sender
>> > > This message came from outside your organization.
>> > > Exercise caution when opening attachments or clicking any
>> > > links.
>> > > |-------------------------------------------------------------------!
>> > >
>> > > Hi,
>> > >
>> > > Thank you very much for your feedbacks. Here is the related PR
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
>> > > .
>> > >
>> > > Regards,
>> > > Nicolas
>> > >
>> > > ________________________________
>> > > From: Otavio Rodolfo Piske <an...@gmail.com>
>> > > Sent: Wednesday, February 9, 2022 10:26
>> > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > Subject: Re: Parallel Build in Camel
>> > >
>> > > !-------------------------------------------------------------------|
>> > > This Message Is From an External Sender
>> > > This message came from outside your organization.
>> > > Exercise caution when opening attachments or clicking any
>> > > links.
>> > > |-------------------------------------------------------------------!
>> > >
>> > > Hi,
>> > >
>> > > I think it would be OK to give it a try. Please, feel free to open a
>> PR
>> > > with this suggestion.
>> > >
>> > > Kind regards
>> > >
>> > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > If the parallel build is something that can be used in the build
>> > > pipeline,
>> > > > why not simply starting by adding the -T option of maven to the
>> > commands
>> > > > launched by the build (
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
>> > > )
>> > > > as short/middle term solution before eventually considering the
>> maven
>> > > > daemon? What do you think of it?
>> > > >
>> > > > Regards,
>> > > > Nicolas
>> > > >
>> > > > ________________________________
>> > > > From: Otavio Rodolfo Piske <an...@gmail.com>
>> > > > Sent: Wednesday, February 9, 2022 10:03
>> > > > To: dev@camel.apache.org <de...@camel.apache.org>
>> > > > Subject: Re: Parallel Build in Camel
>> > > >
>> > > >
>> !-------------------------------------------------------------------|
>> > > > This Message Is From an External Sender
>> > > > This message came from outside your organization.
>> > > > Exercise caution when opening attachments or clicking any
>> > > > links.
>> > > >
>> |-------------------------------------------------------------------!
>> > > >
>> > > > Hi,
>> > > >
>> > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
>> > like
>> > > > to take a closer look at some of the plugins we have on the build to
>> > see
>> > > if
>> > > > there's some easy speed up we could do.
>> > > >
>> > > > As for the Github, I think we depend on 2 things:
>> > > > 1. I believe we need a Github action that contains and runs maven
>> > daemon
>> > > > 2. I believe this action needs to be approved somehow by the ASF. I
>> am
>> > > not
>> > > > sure about this, though ... maybe others in the community know more
>> > about
>> > > > it.
>> > > >
>> > > > Kind regards
>> > > >
>> > > >
>> > > >
>> > > > As a recipient of an email from Talend, your contact personal data
>> will
>> > > be
>> > > > on our systems. Please see our privacy notice (updated August 2020)
>> at
>> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > Otavio R. Piske
>> > >
>> > >
>> >
>> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
>> > >
>> > > As a recipient of an email from Talend, your contact personal data
>> will
>> > be
>> > > on our systems. Please see our privacy notice (updated August 2020) at
>> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > >
>> > >
>> > >
>> > > As a recipient of an email from Talend, your contact personal data
>> will
>> > be
>> > > on our systems. Please see our privacy notice (updated August 2020) at
>> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>> > >
>> > >
>> > >
>> >
>> > --
>> > Otavio R. Piske
>> >
>> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$
>> >
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
>> As a recipient of an email from Talend, your contact personal data will
>> be on our systems. Please see our privacy notice (updated August 2020) at
>> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>>
>>
>>
>> As a recipient of an email from Talend, your contact personal data will
>> be on our systems. Please see our privacy notice. <
>> https://www.talend.com/privacy/>
>>
>>
>>
>
> --
> Otavio R. Piske
> http://orpiske.net
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
Great work, thanks for contributing!
For now I'd keep it as it is, so we can experiment with the parallel runner
on GH. We have a few more things happening on the Jenkins pipeline and it
would be better to keep it as stable as possible for now.
Kind regards
On Wed, Feb 16, 2022 at 5:43 PM Nicolas Filotto <nf...@talend.com> wrote:
> Hi,
>
> I just wanted to let you know that with Otavio, we finally could make the
> Github's runner uses the maven daemon (with 2 threads) as you may have
> seen. It seems to help a little bit since the validation of a PR now takes
> between 45-48 minutes while it used to take more than an hour. I know that
> it is not revolutionary, but I'm afraid that with 2 cores on the runner, it
> will be hard to do much better.
>
> Would it make sense to do the same on the Jenkins' runner?
>
> Regards,
> Nicolas
>
> ________________________________
> From: Nicolas Filotto <nf...@talend.com>
> Sent: Thursday, February 10, 2022 14:10
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi Guillaume,
>
> That's awsome, no I didn't know that option, thanks for sharing, it will
> definitively help.
>
> Actually even the maven daemon as it is now (wondeful job by the way)
> could help.
> Indeed, on my machine even with 2 threads, the command "mvnd clean install
> -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the equivalent
> with maven takes 48 minutes.
>
> I know that we cannot predict the results on the build pipeline based on
> this but at least it sounds like it could be interesting to try it, don't
> you agree?
>
> Regards,
> Nicolas
>
> ________________________________
> From: Guillaume Nodet <gn...@apache.org>
> Sent: Thursday, February 10, 2022 11:24
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Fwiw, i did a lot of work on the build a while ago, but my main goal was to
> optimize subsequent `mvnd -DskipTests` run.
> Ideall, in such cases, no jars should be changed at all, but that's not the
> case completely, especially for maven plugins (try running `ls -ltr
> **/*.jar` after a build). There are pending changes in maven to fix that
> and I plan to push some changes in camel (mostly plugins updates) once
> those are released.
>
> In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
> option to gather statistics on the mojo execution time.
>
> In addition, I've been working a few weeks ago on the
> maven-build-cache-extension [1], which is a powerful extension that
> provides a full per-module build cache. The cache can be configured
> locally and remotely (where the output of a module will be downloaded from
> the remote cache instead of being built locally). It currently requires
> some changes in maven-core that will only be released in maven 3.9.0.
> However, I can try to set up an experimental mvnd branch that would embed
> this extension if that's of any interest.
>
> Guillaume
>
> [1]
> https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$
>
> Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <an...@gmail.com>
> a écrit :
>
> > Thanks for looking into this.
> >
> > On a related note, I think one area where we could investigate potential
> > improvements to the compilation is by looking at the performance of our
> > maven plugins and code generators.
> >
> > For example the camel-endpointdsl module takes a long time to generate
> > sources and build (that one takes almost 2 minutes to run on GH). The
> > camel-componentdsl one takes about 1 minute. I suspect that if we could
> > trace why they are so slow, there may be a route for optimizing the build
> > time (and if we are lucky, that could happen across the whole build).
> >
> > A secondary branch of investigation could be looking at whether we can
> > adjust our build so that we can have it running with "process-classes"
> > target, to avoid the costly process of compressing and installing the
> jars.
> >
> > Kind regards
> >
> > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> > wrote:
> >
> > > Hi again,
> > >
> > > Just to let you know that it appears that there are only 2 cores on the
> > > target server which is obviously not enough to have an impact on the
> > build
> > > time.
> > >
> > > Regards,
> > > Nicolas
> > > ________________________________
> > > From: Nicolas Filotto <nf...@talend.com>
> > > Sent: Wednesday, February 9, 2022 11:33
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > Thank you very much for your feedbacks. Here is the related PR
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > > .
> > >
> > > Regards,
> > > Nicolas
> > >
> > > ________________________________
> > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > Sent: Wednesday, February 9, 2022 10:26
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > I think it would be OK to give it a try. Please, feel free to open a PR
> > > with this suggestion.
> > >
> > > Kind regards
> > >
> > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > If the parallel build is something that can be used in the build
> > > pipeline,
> > > > why not simply starting by adding the -T option of maven to the
> > commands
> > > > launched by the build (
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > > )
> > > > as short/middle term solution before eventually considering the maven
> > > > daemon? What do you think of it?
> > > >
> > > > Regards,
> > > > Nicolas
> > > >
> > > > ________________________________
> > > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > > Sent: Wednesday, February 9, 2022 10:03
> > > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > > Subject: Re: Parallel Build in Camel
> > > >
> > > > !-------------------------------------------------------------------|
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Exercise caution when opening attachments or clicking any
> > > > links.
> > > > |-------------------------------------------------------------------!
> > > >
> > > > Hi,
> > > >
> > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
> > like
> > > > to take a closer look at some of the plugins we have on the build to
> > see
> > > if
> > > > there's some easy speed up we could do.
> > > >
> > > > As for the Github, I think we depend on 2 things:
> > > > 1. I believe we need a Github action that contains and runs maven
> > daemon
> > > > 2. I believe this action needs to be approved somehow by the ASF. I
> am
> > > not
> > > > sure about this, though ... maybe others in the community know more
> > about
> > > > it.
> > > >
> > > > Kind regards
> > > >
> > > >
> > > >
> > > > As a recipient of an email from Talend, your contact personal data
> will
> > > be
> > > > on our systems. Please see our privacy notice (updated August 2020)
> at
> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > > >
> > > >
> > > >
> > >
> > > --
> > > Otavio R. Piske
> > >
> > >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> >
> > --
> > Otavio R. Piske
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$
> >
>
>
> --
> ------------------------
> Guillaume Nodet
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice. <
> https://www.talend.com/privacy/>
>
>
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi,
I just wanted to let you know that with Otavio, we finally could make the Github's runner uses the maven daemon (with 2 threads) as you may have seen. It seems to help a little bit since the validation of a PR now takes between 45-48 minutes while it used to take more than an hour. I know that it is not revolutionary, but I'm afraid that with 2 cores on the runner, it will be hard to do much better.
Would it make sense to do the same on the Jenkins' runner?
Regards,
Nicolas
________________________________
From: Nicolas Filotto <nf...@talend.com>
Sent: Thursday, February 10, 2022 14:10
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hi Guillaume,
That's awsome, no I didn't know that option, thanks for sharing, it will definitively help.
Actually even the maven daemon as it is now (wondeful job by the way) could help.
Indeed, on my machine even with 2 threads, the command "mvnd clean install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the equivalent with maven takes 48 minutes.
I know that we cannot predict the results on the build pipeline based on this but at least it sounds like it could be interesting to try it, don't you agree?
Regards,
Nicolas
________________________________
From: Guillaume Nodet <gn...@apache.org>
Sent: Thursday, February 10, 2022 11:24
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Fwiw, i did a lot of work on the build a while ago, but my main goal was to
optimize subsequent `mvnd -DskipTests` run.
Ideall, in such cases, no jars should be changed at all, but that's not the
case completely, especially for maven plugins (try running `ls -ltr
**/*.jar` after a build). There are pending changes in maven to fix that
and I plan to push some changes in camel (mostly plugins updates) once
those are released.
In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
option to gather statistics on the mojo execution time.
In addition, I've been working a few weeks ago on the
maven-build-cache-extension [1], which is a powerful extension that
provides a full per-module build cache. The cache can be configured
locally and remotely (where the output of a module will be downloaded from
the remote cache instead of being built locally). It currently requires
some changes in maven-core that will only be released in maven 3.9.0.
However, I can try to set up an experimental mvnd branch that would embed
this extension if that's of any interest.
Guillaume
[1] https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$
Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <an...@gmail.com>
a écrit :
> Thanks for looking into this.
>
> On a related note, I think one area where we could investigate potential
> improvements to the compilation is by looking at the performance of our
> maven plugins and code generators.
>
> For example the camel-endpointdsl module takes a long time to generate
> sources and build (that one takes almost 2 minutes to run on GH). The
> camel-componentdsl one takes about 1 minute. I suspect that if we could
> trace why they are so slow, there may be a route for optimizing the build
> time (and if we are lucky, that could happen across the whole build).
>
> A secondary branch of investigation could be looking at whether we can
> adjust our build so that we can have it running with "process-classes"
> target, to avoid the costly process of compressing and installing the jars.
>
> Kind regards
>
> On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
> > Hi again,
> >
> > Just to let you know that it appears that there are only 2 cores on the
> > target server which is obviously not enough to have an impact on the
> build
> > time.
> >
> > Regards,
> > Nicolas
> > ________________________________
> > From: Nicolas Filotto <nf...@talend.com>
> > Sent: Wednesday, February 9, 2022 11:33
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > Thank you very much for your feedbacks. Here is the related PR
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > .
> >
> > Regards,
> > Nicolas
> >
> > ________________________________
> > From: Otavio Rodolfo Piske <an...@gmail.com>
> > Sent: Wednesday, February 9, 2022 10:26
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > I think it would be OK to give it a try. Please, feel free to open a PR
> > with this suggestion.
> >
> > Kind regards
> >
> > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> > wrote:
> >
> > > Hi,
> > >
> > > If the parallel build is something that can be used in the build
> > pipeline,
> > > why not simply starting by adding the -T option of maven to the
> commands
> > > launched by the build (
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > )
> > > as short/middle term solution before eventually considering the maven
> > > daemon? What do you think of it?
> > >
> > > Regards,
> > > Nicolas
> > >
> > > ________________________________
> > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > Sent: Wednesday, February 9, 2022 10:03
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
> like
> > > to take a closer look at some of the plugins we have on the build to
> see
> > if
> > > there's some easy speed up we could do.
> > >
> > > As for the Github, I think we depend on 2 things:
> > > 1. I believe we need a Github action that contains and runs maven
> daemon
> > > 2. I believe this action needs to be approved somehow by the ASF. I am
> > not
> > > sure about this, though ... maybe others in the community know more
> about
> > > it.
> > >
> > > Kind regards
> > >
> > >
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> >
> > --
> > Otavio R. Piske
> >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
>
> --
> Otavio R. Piske
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$
>
--
------------------------
Guillaume Nodet
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice. <https://www.talend.com/privacy/>
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi Guillaume,
That's awsome, no I didn't know that option, thanks for sharing, it will definitively help.
Actually even the maven daemon as it is now (wondeful job by the way) could help.
Indeed, on my machine even with 2 threads, the command "mvnd clean install -Pfastinstall -Psourcecheck -T 2" takes 32 minutes while the equivalent with maven takes 48 minutes.
I know that we cannot predict the results on the build pipeline based on this but at least it sounds like it could be interesting to try it, don't you agree?
Regards,
Nicolas
________________________________
From: Guillaume Nodet <gn...@apache.org>
Sent: Thursday, February 10, 2022 11:24
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Fwiw, i did a lot of work on the build a while ago, but my main goal was to
optimize subsequent `mvnd -DskipTests` run.
Ideall, in such cases, no jars should be changed at all, but that's not the
case completely, especially for maven plugins (try running `ls -ltr
**/*.jar` after a build). There are pending changes in maven to fix that
and I plan to push some changes in camel (mostly plugins updates) once
those are released.
In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
option to gather statistics on the mojo execution time.
In addition, I've been working a few weeks ago on the
maven-build-cache-extension [1], which is a powerful extension that
provides a full per-module build cache. The cache can be configured
locally and remotely (where the output of a module will be downloaded from
the remote cache instead of being built locally). It currently requires
some changes in maven-core that will only be released in maven 3.9.0.
However, I can try to set up an experimental mvnd branch that would embed
this extension if that's of any interest.
Guillaume
[1] https://urldefense.com/v3/__https://github.com/apache/maven-build-cache-extension__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehHmiEG7g$
Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <an...@gmail.com>
a écrit :
> Thanks for looking into this.
>
> On a related note, I think one area where we could investigate potential
> improvements to the compilation is by looking at the performance of our
> maven plugins and code generators.
>
> For example the camel-endpointdsl module takes a long time to generate
> sources and build (that one takes almost 2 minutes to run on GH). The
> camel-componentdsl one takes about 1 minute. I suspect that if we could
> trace why they are so slow, there may be a route for optimizing the build
> time (and if we are lucky, that could happen across the whole build).
>
> A secondary branch of investigation could be looking at whether we can
> adjust our build so that we can have it running with "process-classes"
> target, to avoid the costly process of compressing and installing the jars.
>
> Kind regards
>
> On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
> > Hi again,
> >
> > Just to let you know that it appears that there are only 2 cores on the
> > target server which is obviously not enough to have an impact on the
> build
> > time.
> >
> > Regards,
> > Nicolas
> > ________________________________
> > From: Nicolas Filotto <nf...@talend.com>
> > Sent: Wednesday, February 9, 2022 11:33
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > Thank you very much for your feedbacks. Here is the related PR
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > .
> >
> > Regards,
> > Nicolas
> >
> > ________________________________
> > From: Otavio Rodolfo Piske <an...@gmail.com>
> > Sent: Wednesday, February 9, 2022 10:26
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > I think it would be OK to give it a try. Please, feel free to open a PR
> > with this suggestion.
> >
> > Kind regards
> >
> > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> > wrote:
> >
> > > Hi,
> > >
> > > If the parallel build is something that can be used in the build
> > pipeline,
> > > why not simply starting by adding the -T option of maven to the
> commands
> > > launched by the build (
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > )
> > > as short/middle term solution before eventually considering the maven
> > > daemon? What do you think of it?
> > >
> > > Regards,
> > > Nicolas
> > >
> > > ________________________________
> > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > Sent: Wednesday, February 9, 2022 10:03
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
> like
> > > to take a closer look at some of the plugins we have on the build to
> see
> > if
> > > there's some easy speed up we could do.
> > >
> > > As for the Github, I think we depend on 2 things:
> > > 1. I believe we need a Github action that contains and runs maven
> daemon
> > > 2. I believe this action needs to be approved somehow by the ASF. I am
> > not
> > > sure about this, though ... maybe others in the community know more
> about
> > > it.
> > >
> > > Kind regards
> > >
> > >
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> >
> > --
> > Otavio R. Piske
> >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
>
> --
> Otavio R. Piske
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HCRQGPM2NNJHbYh_rZneQbVDhQ9sp14RTKOq3aj88pSnBMTttsExKdehQC12Kv4$
>
--
------------------------
Guillaume Nodet
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by Guillaume Nodet <gn...@apache.org>.
Le jeu. 10 févr. 2022 à 12:33, Otavio Rodolfo Piske <an...@gmail.com>
a écrit :
> This sounds great! Looking forward to trying it out.
>
I'll try to set up something in the coming days.
>
> Thanks for the tip about the " -Dmvnd.buildTime"!
>
> Looking at the build time report from maven daemon: do I understand
> correctly that the last part of the report is the totals for each plugin?
>
Yes
>
> Looking at what we have, it seems camel-package-maven-plugin seems to be
> one of the problematic ones.
>
For this plugin, it should be possible to look at the timestamp of the
input files and compare it with the generated one.
The difficult part may be to somehow extract all the input files, but this
looks like a good possible improvement at first glance.
What I did optimize is the fact that the output is not overwritten if it
has not changed, it at least allows the compiler to skip rebuilding all
files, so it did reduce the build time a lot.
However, I'm not sure if it can be optimized from a clean build. A more in
depth analysis with a profiler may help...
Guillaume
>
> Kind regards
>
>
> On Thu, Feb 10, 2022 at 11:24 AM Guillaume Nodet <gn...@apache.org>
> wrote:
>
> > Fwiw, i did a lot of work on the build a while ago, but my main goal was
> to
> > optimize subsequent `mvnd -DskipTests` run.
> > Ideall, in such cases, no jars should be changed at all, but that's not
> the
> > case completely, especially for maven plugins (try running `ls -ltr
> > **/*.jar` after a build). There are pending changes in maven to fix that
> > and I plan to push some changes in camel (mostly plugins updates) once
> > those are released.
> >
> > In case you don't know the option, you can use the `mvnd
> -Dmvnd.buildTime`
> > option to gather statistics on the mojo execution time.
> >
> > In addition, I've been working a few weeks ago on the
> > maven-build-cache-extension [1], which is a powerful extension that
> > provides a full per-module build cache. The cache can be configured
> > locally and remotely (where the output of a module will be downloaded
> from
> > the remote cache instead of being built locally). It currently requires
> > some changes in maven-core that will only be released in maven 3.9.0.
> > However, I can try to set up an experimental mvnd branch that would embed
> > this extension if that's of any interest.
> >
> > Guillaume
> >
> > [1] https://github.com/apache/maven-build-cache-extension
> >
> > Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <
> angusyoung@gmail.com>
> > a écrit :
> >
> > > Thanks for looking into this.
> > >
> > > On a related note, I think one area where we could investigate
> potential
> > > improvements to the compilation is by looking at the performance of our
> > > maven plugins and code generators.
> > >
> > > For example the camel-endpointdsl module takes a long time to generate
> > > sources and build (that one takes almost 2 minutes to run on GH). The
> > > camel-componentdsl one takes about 1 minute. I suspect that if we could
> > > trace why they are so slow, there may be a route for optimizing the
> build
> > > time (and if we are lucky, that could happen across the whole build).
> > >
> > > A secondary branch of investigation could be looking at whether we can
> > > adjust our build so that we can have it running with "process-classes"
> > > target, to avoid the costly process of compressing and installing the
> > jars.
> > >
> > > Kind regards
> > >
> > > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> > > wrote:
> > >
> > > > Hi again,
> > > >
> > > > Just to let you know that it appears that there are only 2 cores on
> the
> > > > target server which is obviously not enough to have an impact on the
> > > build
> > > > time.
> > > >
> > > > Regards,
> > > > Nicolas
> > > > ________________________________
> > > > From: Nicolas Filotto <nf...@talend.com>
> > > > Sent: Wednesday, February 9, 2022 11:33
> > > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > > Subject: Re: Parallel Build in Camel
> > > >
> > > > !-------------------------------------------------------------------|
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Exercise caution when opening attachments or clicking any
> > > > links.
> > > > |-------------------------------------------------------------------!
> > > >
> > > > Hi,
> > > >
> > > > Thank you very much for your feedbacks. Here is the related PR
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > > > .
> > > >
> > > > Regards,
> > > > Nicolas
> > > >
> > > > ________________________________
> > > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > > Sent: Wednesday, February 9, 2022 10:26
> > > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > > Subject: Re: Parallel Build in Camel
> > > >
> > > > !-------------------------------------------------------------------|
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Exercise caution when opening attachments or clicking any
> > > > links.
> > > > |-------------------------------------------------------------------!
> > > >
> > > > Hi,
> > > >
> > > > I think it would be OK to give it a try. Please, feel free to open a
> PR
> > > > with this suggestion.
> > > >
> > > > Kind regards
> > > >
> > > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nfilotto@talend.com
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > If the parallel build is something that can be used in the build
> > > > pipeline,
> > > > > why not simply starting by adding the -T option of maven to the
> > > commands
> > > > > launched by the build (
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > > > )
> > > > > as short/middle term solution before eventually considering the
> maven
> > > > > daemon? What do you think of it?
> > > > >
> > > > > Regards,
> > > > > Nicolas
> > > > >
> > > > > ________________________________
> > > > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > > > Sent: Wednesday, February 9, 2022 10:03
> > > > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > > > Subject: Re: Parallel Build in Camel
> > > > >
> > > > >
> !-------------------------------------------------------------------|
> > > > > This Message Is From an External Sender
> > > > > This message came from outside your organization.
> > > > > Exercise caution when opening attachments or clicking any
> > > > > links.
> > > > >
> |-------------------------------------------------------------------!
> > > > >
> > > > > Hi,
> > > > >
> > > > > Yeah, that helps a bit, although it's still quite a lot. One day
> I'd
> > > like
> > > > > to take a closer look at some of the plugins we have on the build
> to
> > > see
> > > > if
> > > > > there's some easy speed up we could do.
> > > > >
> > > > > As for the Github, I think we depend on 2 things:
> > > > > 1. I believe we need a Github action that contains and runs maven
> > > daemon
> > > > > 2. I believe this action needs to be approved somehow by the ASF. I
> > am
> > > > not
> > > > > sure about this, though ... maybe others in the community know more
> > > about
> > > > > it.
> > > > >
> > > > > Kind regards
> > > > >
> > > > >
> > > > >
> > > > > As a recipient of an email from Talend, your contact personal data
> > will
> > > > be
> > > > > on our systems. Please see our privacy notice (updated August 2020)
> > at
> > > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Otavio R. Piske
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> > > >
> > > > As a recipient of an email from Talend, your contact personal data
> will
> > > be
> > > > on our systems. Please see our privacy notice (updated August 2020)
> at
> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > > >
> > > >
> > > >
> > > > As a recipient of an email from Talend, your contact personal data
> will
> > > be
> > > > on our systems. Please see our privacy notice (updated August 2020)
> at
> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > > >
> > > >
> > > >
> > >
> > > --
> > > Otavio R. Piske
> > > http://orpiske.net
> > >
> >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
>
>
> --
> Otavio R. Piske
> http://orpiske.net
>
--
------------------------
Guillaume Nodet
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
This sounds great! Looking forward to trying it out.
Thanks for the tip about the " -Dmvnd.buildTime"!
Looking at the build time report from maven daemon: do I understand
correctly that the last part of the report is the totals for each plugin?
Looking at what we have, it seems camel-package-maven-plugin seems to be
one of the problematic ones.
Kind regards
On Thu, Feb 10, 2022 at 11:24 AM Guillaume Nodet <gn...@apache.org> wrote:
> Fwiw, i did a lot of work on the build a while ago, but my main goal was to
> optimize subsequent `mvnd -DskipTests` run.
> Ideall, in such cases, no jars should be changed at all, but that's not the
> case completely, especially for maven plugins (try running `ls -ltr
> **/*.jar` after a build). There are pending changes in maven to fix that
> and I plan to push some changes in camel (mostly plugins updates) once
> those are released.
>
> In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
> option to gather statistics on the mojo execution time.
>
> In addition, I've been working a few weeks ago on the
> maven-build-cache-extension [1], which is a powerful extension that
> provides a full per-module build cache. The cache can be configured
> locally and remotely (where the output of a module will be downloaded from
> the remote cache instead of being built locally). It currently requires
> some changes in maven-core that will only be released in maven 3.9.0.
> However, I can try to set up an experimental mvnd branch that would embed
> this extension if that's of any interest.
>
> Guillaume
>
> [1] https://github.com/apache/maven-build-cache-extension
>
> Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <an...@gmail.com>
> a écrit :
>
> > Thanks for looking into this.
> >
> > On a related note, I think one area where we could investigate potential
> > improvements to the compilation is by looking at the performance of our
> > maven plugins and code generators.
> >
> > For example the camel-endpointdsl module takes a long time to generate
> > sources and build (that one takes almost 2 minutes to run on GH). The
> > camel-componentdsl one takes about 1 minute. I suspect that if we could
> > trace why they are so slow, there may be a route for optimizing the build
> > time (and if we are lucky, that could happen across the whole build).
> >
> > A secondary branch of investigation could be looking at whether we can
> > adjust our build so that we can have it running with "process-classes"
> > target, to avoid the costly process of compressing and installing the
> jars.
> >
> > Kind regards
> >
> > On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> > wrote:
> >
> > > Hi again,
> > >
> > > Just to let you know that it appears that there are only 2 cores on the
> > > target server which is obviously not enough to have an impact on the
> > build
> > > time.
> > >
> > > Regards,
> > > Nicolas
> > > ________________________________
> > > From: Nicolas Filotto <nf...@talend.com>
> > > Sent: Wednesday, February 9, 2022 11:33
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > Thank you very much for your feedbacks. Here is the related PR
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > > .
> > >
> > > Regards,
> > > Nicolas
> > >
> > > ________________________________
> > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > Sent: Wednesday, February 9, 2022 10:26
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > I think it would be OK to give it a try. Please, feel free to open a PR
> > > with this suggestion.
> > >
> > > Kind regards
> > >
> > > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > If the parallel build is something that can be used in the build
> > > pipeline,
> > > > why not simply starting by adding the -T option of maven to the
> > commands
> > > > launched by the build (
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > > )
> > > > as short/middle term solution before eventually considering the maven
> > > > daemon? What do you think of it?
> > > >
> > > > Regards,
> > > > Nicolas
> > > >
> > > > ________________________________
> > > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > > Sent: Wednesday, February 9, 2022 10:03
> > > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > > Subject: Re: Parallel Build in Camel
> > > >
> > > > !-------------------------------------------------------------------|
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Exercise caution when opening attachments or clicking any
> > > > links.
> > > > |-------------------------------------------------------------------!
> > > >
> > > > Hi,
> > > >
> > > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
> > like
> > > > to take a closer look at some of the plugins we have on the build to
> > see
> > > if
> > > > there's some easy speed up we could do.
> > > >
> > > > As for the Github, I think we depend on 2 things:
> > > > 1. I believe we need a Github action that contains and runs maven
> > daemon
> > > > 2. I believe this action needs to be approved somehow by the ASF. I
> am
> > > not
> > > > sure about this, though ... maybe others in the community know more
> > about
> > > > it.
> > > >
> > > > Kind regards
> > > >
> > > >
> > > >
> > > > As a recipient of an email from Talend, your contact personal data
> will
> > > be
> > > > on our systems. Please see our privacy notice (updated August 2020)
> at
> > > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > > >
> > > >
> > > >
> > >
> > > --
> > > Otavio R. Piske
> > >
> > >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> >
> > --
> > Otavio R. Piske
> > http://orpiske.net
> >
>
>
> --
> ------------------------
> Guillaume Nodet
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Guillaume Nodet <gn...@apache.org>.
Fwiw, i did a lot of work on the build a while ago, but my main goal was to
optimize subsequent `mvnd -DskipTests` run.
Ideall, in such cases, no jars should be changed at all, but that's not the
case completely, especially for maven plugins (try running `ls -ltr
**/*.jar` after a build). There are pending changes in maven to fix that
and I plan to push some changes in camel (mostly plugins updates) once
those are released.
In case you don't know the option, you can use the `mvnd -Dmvnd.buildTime`
option to gather statistics on the mojo execution time.
In addition, I've been working a few weeks ago on the
maven-build-cache-extension [1], which is a powerful extension that
provides a full per-module build cache. The cache can be configured
locally and remotely (where the output of a module will be downloaded from
the remote cache instead of being built locally). It currently requires
some changes in maven-core that will only be released in maven 3.9.0.
However, I can try to set up an experimental mvnd branch that would embed
this extension if that's of any interest.
Guillaume
[1] https://github.com/apache/maven-build-cache-extension
Le jeu. 10 févr. 2022 à 09:25, Otavio Rodolfo Piske <an...@gmail.com>
a écrit :
> Thanks for looking into this.
>
> On a related note, I think one area where we could investigate potential
> improvements to the compilation is by looking at the performance of our
> maven plugins and code generators.
>
> For example the camel-endpointdsl module takes a long time to generate
> sources and build (that one takes almost 2 minutes to run on GH). The
> camel-componentdsl one takes about 1 minute. I suspect that if we could
> trace why they are so slow, there may be a route for optimizing the build
> time (and if we are lucky, that could happen across the whole build).
>
> A secondary branch of investigation could be looking at whether we can
> adjust our build so that we can have it running with "process-classes"
> target, to avoid the costly process of compressing and installing the jars.
>
> Kind regards
>
> On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
> > Hi again,
> >
> > Just to let you know that it appears that there are only 2 cores on the
> > target server which is obviously not enough to have an impact on the
> build
> > time.
> >
> > Regards,
> > Nicolas
> > ________________________________
> > From: Nicolas Filotto <nf...@talend.com>
> > Sent: Wednesday, February 9, 2022 11:33
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > Thank you very much for your feedbacks. Here is the related PR
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> > .
> >
> > Regards,
> > Nicolas
> >
> > ________________________________
> > From: Otavio Rodolfo Piske <an...@gmail.com>
> > Sent: Wednesday, February 9, 2022 10:26
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > I think it would be OK to give it a try. Please, feel free to open a PR
> > with this suggestion.
> >
> > Kind regards
> >
> > On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> > wrote:
> >
> > > Hi,
> > >
> > > If the parallel build is something that can be used in the build
> > pipeline,
> > > why not simply starting by adding the -T option of maven to the
> commands
> > > launched by the build (
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> > )
> > > as short/middle term solution before eventually considering the maven
> > > daemon? What do you think of it?
> > >
> > > Regards,
> > > Nicolas
> > >
> > > ________________________________
> > > From: Otavio Rodolfo Piske <an...@gmail.com>
> > > Sent: Wednesday, February 9, 2022 10:03
> > > To: dev@camel.apache.org <de...@camel.apache.org>
> > > Subject: Re: Parallel Build in Camel
> > >
> > > !-------------------------------------------------------------------|
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Exercise caution when opening attachments or clicking any
> > > links.
> > > |-------------------------------------------------------------------!
> > >
> > > Hi,
> > >
> > > Yeah, that helps a bit, although it's still quite a lot. One day I'd
> like
> > > to take a closer look at some of the plugins we have on the build to
> see
> > if
> > > there's some easy speed up we could do.
> > >
> > > As for the Github, I think we depend on 2 things:
> > > 1. I believe we need a Github action that contains and runs maven
> daemon
> > > 2. I believe this action needs to be approved somehow by the ASF. I am
> > not
> > > sure about this, though ... maybe others in the community know more
> about
> > > it.
> > >
> > > Kind regards
> > >
> > >
> > >
> > > As a recipient of an email from Talend, your contact personal data will
> > be
> > > on our systems. Please see our privacy notice (updated August 2020) at
> > > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> > >
> > >
> > >
> >
> > --
> > Otavio R. Piske
> >
> >
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
>
> --
> Otavio R. Piske
> http://orpiske.net
>
--
------------------------
Guillaume Nodet
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
Thanks for looking into this.
On a related note, I think one area where we could investigate potential
improvements to the compilation is by looking at the performance of our
maven plugins and code generators.
For example the camel-endpointdsl module takes a long time to generate
sources and build (that one takes almost 2 minutes to run on GH). The
camel-componentdsl one takes about 1 minute. I suspect that if we could
trace why they are so slow, there may be a route for optimizing the build
time (and if we are lucky, that could happen across the whole build).
A secondary branch of investigation could be looking at whether we can
adjust our build so that we can have it running with "process-classes"
target, to avoid the costly process of compressing and installing the jars.
Kind regards
On Wed, Feb 9, 2022 at 2:41 PM Nicolas Filotto <nf...@talend.com> wrote:
> Hi again,
>
> Just to let you know that it appears that there are only 2 cores on the
> target server which is obviously not enough to have an impact on the build
> time.
>
> Regards,
> Nicolas
> ________________________________
> From: Nicolas Filotto <nf...@talend.com>
> Sent: Wednesday, February 9, 2022 11:33
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi,
>
> Thank you very much for your feedbacks. Here is the related PR
> https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$
> .
>
> Regards,
> Nicolas
>
> ________________________________
> From: Otavio Rodolfo Piske <an...@gmail.com>
> Sent: Wednesday, February 9, 2022 10:26
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi,
>
> I think it would be OK to give it a try. Please, feel free to open a PR
> with this suggestion.
>
> Kind regards
>
> On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com>
> wrote:
>
> > Hi,
> >
> > If the parallel build is something that can be used in the build
> pipeline,
> > why not simply starting by adding the -T option of maven to the commands
> > launched by the build (
> >
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$
> )
> > as short/middle term solution before eventually considering the maven
> > daemon? What do you think of it?
> >
> > Regards,
> > Nicolas
> >
> > ________________________________
> > From: Otavio Rodolfo Piske <an...@gmail.com>
> > Sent: Wednesday, February 9, 2022 10:03
> > To: dev@camel.apache.org <de...@camel.apache.org>
> > Subject: Re: Parallel Build in Camel
> >
> > !-------------------------------------------------------------------|
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Exercise caution when opening attachments or clicking any
> > links.
> > |-------------------------------------------------------------------!
> >
> > Hi,
> >
> > Yeah, that helps a bit, although it's still quite a lot. One day I'd like
> > to take a closer look at some of the plugins we have on the build to see
> if
> > there's some easy speed up we could do.
> >
> > As for the Github, I think we depend on 2 things:
> > 1. I believe we need a Github action that contains and runs maven daemon
> > 2. I believe this action needs to be approved somehow by the ASF. I am
> not
> > sure about this, though ... maybe others in the community know more about
> > it.
> >
> > Kind regards
> >
> >
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
>
> --
> Otavio R. Piske
>
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi again,
Just to let you know that it appears that there are only 2 cores on the target server which is obviously not enough to have an impact on the build time.
Regards,
Nicolas
________________________________
From: Nicolas Filotto <nf...@talend.com>
Sent: Wednesday, February 9, 2022 11:33
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hi,
Thank you very much for your feedbacks. Here is the related PR https://urldefense.com/v3/__https://github.com/apache/camel/pull/6907__;!!CiXD_PY!C-yfZjEVUOAkwpdj1K9URzQDT9hlsdcr4PMcsL2byb6MCaomvtqUJvVUoIKNkHE$ .
Regards,
Nicolas
________________________________
From: Otavio Rodolfo Piske <an...@gmail.com>
Sent: Wednesday, February 9, 2022 10:26
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hi,
I think it would be OK to give it a try. Please, feel free to open a PR
with this suggestion.
Kind regards
On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com> wrote:
> Hi,
>
> If the parallel build is something that can be used in the build pipeline,
> why not simply starting by adding the -T option of maven to the commands
> launched by the build (
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$ )
> as short/middle term solution before eventually considering the maven
> daemon? What do you think of it?
>
> Regards,
> Nicolas
>
> ________________________________
> From: Otavio Rodolfo Piske <an...@gmail.com>
> Sent: Wednesday, February 9, 2022 10:03
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi,
>
> Yeah, that helps a bit, although it's still quite a lot. One day I'd like
> to take a closer look at some of the plugins we have on the build to see if
> there's some easy speed up we could do.
>
> As for the Github, I think we depend on 2 things:
> 1. I believe we need a Github action that contains and runs maven daemon
> 2. I believe this action needs to be approved somehow by the ASF. I am not
> sure about this, though ... maybe others in the community know more about
> it.
>
> Kind regards
>
>
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi,
Thank you very much for your feedbacks. Here is the related PR https://github.com/apache/camel/pull/6907.
Regards,
Nicolas
________________________________
From: Otavio Rodolfo Piske <an...@gmail.com>
Sent: Wednesday, February 9, 2022 10:26
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hi,
I think it would be OK to give it a try. Please, feel free to open a PR
with this suggestion.
Kind regards
On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com> wrote:
> Hi,
>
> If the parallel build is something that can be used in the build pipeline,
> why not simply starting by adding the -T option of maven to the commands
> launched by the build (
> https://urldefense.com/v3/__https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml*L39__;Iw!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33k5dXjH0$ )
> as short/middle term solution before eventually considering the maven
> daemon? What do you think of it?
>
> Regards,
> Nicolas
>
> ________________________________
> From: Otavio Rodolfo Piske <an...@gmail.com>
> Sent: Wednesday, February 9, 2022 10:03
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi,
>
> Yeah, that helps a bit, although it's still quite a lot. One day I'd like
> to take a closer look at some of the plugins we have on the build to see if
> there's some easy speed up we could do.
>
> As for the Github, I think we depend on 2 things:
> 1. I believe we need a Github action that contains and runs maven daemon
> 2. I believe this action needs to be approved somehow by the ASF. I am not
> sure about this, though ... maybe others in the community know more about
> it.
>
> Kind regards
>
>
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!EMmp3maGNIUrNLCyD1a__sEjX6IOh6rg2uTID5to5gtnpo19lPAOuN33LLnzx4c$
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
Hi,
I think it would be OK to give it a try. Please, feel free to open a PR
with this suggestion.
Kind regards
On Wed, Feb 9, 2022 at 10:22 AM Nicolas Filotto <nf...@talend.com> wrote:
> Hi,
>
> If the parallel build is something that can be used in the build pipeline,
> why not simply starting by adding the -T option of maven to the commands
> launched by the build (
> https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml#L39)
> as short/middle term solution before eventually considering the maven
> daemon? What do you think of it?
>
> Regards,
> Nicolas
>
> ________________________________
> From: Otavio Rodolfo Piske <an...@gmail.com>
> Sent: Wednesday, February 9, 2022 10:03
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hi,
>
> Yeah, that helps a bit, although it's still quite a lot. One day I'd like
> to take a closer look at some of the plugins we have on the build to see if
> there's some easy speed up we could do.
>
> As for the Github, I think we depend on 2 things:
> 1. I believe we need a Github action that contains and runs maven daemon
> 2. I believe this action needs to be approved somehow by the ASF. I am not
> sure about this, though ... maybe others in the community know more about
> it.
>
> Kind regards
>
>
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi,
If the parallel build is something that can be used in the build pipeline, why not simply starting by adding the -T option of maven to the commands launched by the build (https://github.com/apache/camel/blob/main/.github/workflows/master-pr-build.yml#L39) as short/middle term solution before eventually considering the maven daemon? What do you think of it?
Regards,
Nicolas
________________________________
From: Otavio Rodolfo Piske <an...@gmail.com>
Sent: Wednesday, February 9, 2022 10:03
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hi,
Yeah, that helps a bit, although it's still quite a lot. One day I'd like
to take a closer look at some of the plugins we have on the build to see if
there's some easy speed up we could do.
As for the Github, I think we depend on 2 things:
1. I believe we need a Github action that contains and runs maven daemon
2. I believe this action needs to be approved somehow by the ASF. I am not
sure about this, though ... maybe others in the community know more about
it.
Kind regards
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
Hi,
Yeah, that helps a bit, although it's still quite a lot. One day I'd like
to take a closer look at some of the plugins we have on the build to see if
there's some easy speed up we could do.
As for the Github, I think we depend on 2 things:
1. I believe we need a Github action that contains and runs maven daemon
2. I believe this action needs to be approved somehow by the ASF. I am not
sure about this, though ... maybe others in the community know more about
it.
Kind regards
On Tue, Feb 8, 2022 at 6:48 PM Nicolas Filotto <nf...@talend.com> wrote:
> Hi
>
> That's a very nice tool thanks for sharing! It is indeed even faster, it
> takes "only" 21 minutes on my machine which is much better.
>
> As it is significantly faster and generally reliable, any reason why it is
> not used in the build pipeline? At least to validate the PRs when the tests
> are skipped
>
> Regards,
> Nicolas
>
> ________________________________
> From: Otavio Rodolfo Piske <an...@gmail.com>
> Sent: Tuesday, February 8, 2022 17:34
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Parallel Build in Camel
>
> !-------------------------------------------------------------------|
> This Message Is From an External Sender
> This message came from outside your organization.
> Exercise caution when opening attachments or clicking any
> links.
> |-------------------------------------------------------------------!
>
> Hello,
>
> I have been using parallel builds with Camel and all it's sub-projects
> regularly. On some (rare) occasions I find a problem or two, but it's
> usually solved with rebuilding or running a non-parallel build.
>
> In my case, I build using Maven Daemon [1] and it has proven to be very
> stable in terms of reliability. A cached build of Camel takes about 1
> minute on my Desktop (Ryzen 9 3900XT 12 cores / 24 threads, 32Gb RAM). I
> don't have the precise numbers, but a clean build with Maven Daemon takes
> about 9 minutes. On my MBP with a similar configuration to yours, a cached
> build is around 3-4 minutes and a non-cached is between 14 to 18 minutes.
> Not great, but not terrible either.
>
> 1.
> https://urldefense.com/v3/__https://github.com/apache/maven-mvnd__;!!CiXD_PY!HVOrEMl7_mFR4dV-jkTqvqa6h1P9X-qJRX7MyRs6WeOMqhjt6g6J-jfihrMwRpE$
>
> Kind regards
>
> On Tue, Feb 8, 2022 at 5:27 PM Nicolas Filotto <nf...@talend.com>
> wrote:
>
> > Hi Cameleers,
> >
> > I was wondering if it was safe to use the parallel build of maven at
> least
> > to compile camel faster. Are you aware of any issues with it?
> >
> > I'm asking that because on my local machine (MBP 2,6 GHz Intel Core i7
> > with 6 cores), launching:
> >
> > * mvn clean install -Pfastinstall -Psourcecheck - takes about 50
> > minutes
> > * mvn -T 1C clean install -Pfastinstall -Psourcecheck - takes about
> 30
> > minutes
> >
> > So, it is not revolutionary, but it can still be helpful if it is
> > reliable/safe enough.
> >
> > Thanks in advance for your answers,
> > Nicolas
> >
> > As a recipient of an email from Talend, your contact personal data will
> be
> > on our systems. Please see our privacy notice (updated August 2020) at
> > Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
> >
> >
> >
>
> --
> Otavio R. Piske
>
> https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HVOrEMl7_mFR4dV-jkTqvqa6h1P9X-qJRX7MyRs6WeOMqhjt6g6J-jfiKrdfAgY$
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
http://orpiske.net
Re: Parallel Build in Camel
Posted by Nicolas Filotto <nf...@talend.com>.
Hi
That's a very nice tool thanks for sharing! It is indeed even faster, it takes "only" 21 minutes on my machine which is much better.
As it is significantly faster and generally reliable, any reason why it is not used in the build pipeline? At least to validate the PRs when the tests are skipped
Regards,
Nicolas
________________________________
From: Otavio Rodolfo Piske <an...@gmail.com>
Sent: Tuesday, February 8, 2022 17:34
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Parallel Build in Camel
!-------------------------------------------------------------------|
This Message Is From an External Sender
This message came from outside your organization.
Exercise caution when opening attachments or clicking any
links.
|-------------------------------------------------------------------!
Hello,
I have been using parallel builds with Camel and all it's sub-projects
regularly. On some (rare) occasions I find a problem or two, but it's
usually solved with rebuilding or running a non-parallel build.
In my case, I build using Maven Daemon [1] and it has proven to be very
stable in terms of reliability. A cached build of Camel takes about 1
minute on my Desktop (Ryzen 9 3900XT 12 cores / 24 threads, 32Gb RAM). I
don't have the precise numbers, but a clean build with Maven Daemon takes
about 9 minutes. On my MBP with a similar configuration to yours, a cached
build is around 3-4 minutes and a non-cached is between 14 to 18 minutes.
Not great, but not terrible either.
1. https://urldefense.com/v3/__https://github.com/apache/maven-mvnd__;!!CiXD_PY!HVOrEMl7_mFR4dV-jkTqvqa6h1P9X-qJRX7MyRs6WeOMqhjt6g6J-jfihrMwRpE$
Kind regards
On Tue, Feb 8, 2022 at 5:27 PM Nicolas Filotto <nf...@talend.com> wrote:
> Hi Cameleers,
>
> I was wondering if it was safe to use the parallel build of maven at least
> to compile camel faster. Are you aware of any issues with it?
>
> I'm asking that because on my local machine (MBP 2,6 GHz Intel Core i7
> with 6 cores), launching:
>
> * mvn clean install -Pfastinstall -Psourcecheck - takes about 50
> minutes
> * mvn -T 1C clean install -Pfastinstall -Psourcecheck - takes about 30
> minutes
>
> So, it is not revolutionary, but it can still be helpful if it is
> reliable/safe enough.
>
> Thanks in advance for your answers,
> Nicolas
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
https://urldefense.com/v3/__http://orpiske.net__;!!CiXD_PY!HVOrEMl7_mFR4dV-jkTqvqa6h1P9X-qJRX7MyRs6WeOMqhjt6g6J-jfiKrdfAgY$
As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice (updated August 2020) at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
Re: Parallel Build in Camel
Posted by Otavio Rodolfo Piske <an...@gmail.com>.
Hello,
I have been using parallel builds with Camel and all it's sub-projects
regularly. On some (rare) occasions I find a problem or two, but it's
usually solved with rebuilding or running a non-parallel build.
In my case, I build using Maven Daemon [1] and it has proven to be very
stable in terms of reliability. A cached build of Camel takes about 1
minute on my Desktop (Ryzen 9 3900XT 12 cores / 24 threads, 32Gb RAM). I
don't have the precise numbers, but a clean build with Maven Daemon takes
about 9 minutes. On my MBP with a similar configuration to yours, a cached
build is around 3-4 minutes and a non-cached is between 14 to 18 minutes.
Not great, but not terrible either.
1. https://github.com/apache/maven-mvnd
Kind regards
On Tue, Feb 8, 2022 at 5:27 PM Nicolas Filotto <nf...@talend.com> wrote:
> Hi Cameleers,
>
> I was wondering if it was safe to use the parallel build of maven at least
> to compile camel faster. Are you aware of any issues with it?
>
> I'm asking that because on my local machine (MBP 2,6 GHz Intel Core i7
> with 6 cores), launching:
>
> * mvn clean install -Pfastinstall -Psourcecheck - takes about 50
> minutes
> * mvn -T 1C clean install -Pfastinstall -Psourcecheck - takes about 30
> minutes
>
> So, it is not revolutionary, but it can still be helpful if it is
> reliable/safe enough.
>
> Thanks in advance for your answers,
> Nicolas
>
> As a recipient of an email from Talend, your contact personal data will be
> on our systems. Please see our privacy notice (updated August 2020) at
> Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>
>
>
>
--
Otavio R. Piske
http://orpiske.net