You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Ignasi Barrera <ig...@gmail.com> on 2014/04/28 12:55:12 UTC
Speed up jclouds builds
Hi!
Just a thought about our current builds and if we can speed them up.
Maven has the "-T" option to set the number of threads that are going
to run the builds, and we could use it to run them concurrently.
Something like "-T1C" will spawn one thread per core, and run those
builds in parallel.
We have many independent builds (many providers/apis don't depend on
each other) so that could speed up our existing builds, in general.
Has someone tried to play with the "-T" maven parameter when building jclouds?
Do you know which kind of instances do we have in Dev@cloud, and if we
could set that parameter?
Ignasi
Re: Speed up jclouds builds
Posted by Andrew Phillips <ap...@qrmedia.com>.
Added "-T 2C" to "jclouds-pull-requests" but *not* to
"jclouds-java-7-pull-requests", so we should be able to see what (if
any) the difference is.
ap
Re: Speed up jclouds builds
Posted by Everett Toews <ev...@RACKSPACE.COM>.
Let’s just see what happens.
Everett
On Apr 29, 2014, at 12:51 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
> What data would we need on the DEV@cloud builds to start experimenting here? Or should we simply add -T 2C or so to the PR builds and see what happens?
>
> ap
Re: Speed up jclouds builds
Posted by Andrew Phillips <ap...@qrmedia.com>.
What data would we need on the DEV@cloud builds to start experimenting
here? Or should we simply add -T 2C or so to the PR builds and see
what happens?
ap
Re: Speed up jclouds builds
Posted by Jeremy Daggett <je...@RACKSPACE.COM>.
Thanks for bringing this up Ignasi!
In a previous life, I was a build engineer on a fairly large project where
the build times could take anywhere from 60-90 min. Fortunately, I was
successful in reducing build times to around 25% of the time they were
taking. It improved developer productivity immensely and developers
experienced happiness they never knew existed. ;)
Yesterday, I grabbed a few more data points using the Maven ³-T" parameter
when increasing the number of threads. I ran and timed the builds on my
MBP 2.4GHz Intel Core i7 / 8 GB with the following command:
$ time mvn clean install -T x
Where Œx' is the number of threads...
* Build Results - jclouds/clouds
mvn clean install 792.45s user 50.02s system 153% cpu 9:10.01 total
mvn clean install -T 5 1157.68s user 71.89s system 501% cpu 4:05.40 total
mvn clean install -T 8 1142.30s user 74.63s system 505% cpu 4:00.59 total
mvn clean install -T 10 1143.61s user 73.77s system 507% cpu 3:59.75 total
mvn clean install -T 15 1146.27s user 74.06s system 508% cpu 3:59.84 total
Avg Memory: ~190M
* Build Results - jclouds/jclouds-labs-openstack
mvn clean install 128.90s user 6.75s system 130% cpu 1:43.60 total
mvn clean install -T 5 197.95s user 9.27s system 269% cpu 1:16.84 total
mvn clean install -T 10 214.68s user 9.59s system 277% cpu 1:20.84 total
Avg Memory: ~53M
Memory is pretty consistent and setting the number of threads to 5 cuts
the jclouds/jclouds time dramatically. I know that it can improve even
more with a little analysis.
/jd
On 4/29/14, 6:05 AM, "Ignasi Barrera" <ig...@gmail.com> wrote:
>I just discovered the option and that's why I asked :)
>
>A local build of the "jclouds" repo without and with it shows the time
>you can save (this is in a machine with 4 cores):
>
>$ mvn clean install
>Total time: 08:07 min
>
>$ mvn clean install -T1C
>Total time: 03:51 min (Wall Clock)
>
>It takes half the time. Don't know if with the instances we have we'll
>see some improvement, but perhaps is worth trying. It would be cool to
>reduce the PR build time.
>
>On 28 April 2014 17:00, Andrew Phillips <ap...@qrmedia.com> wrote:
>>> Has someone tried to play with the "-T" maven parameter when building
>>> jclouds?
>>> Do you know which kind of instances do we have in Dev@cloud, and if we
>>> could set that parameter?
>>
>>
>> I recall having tried it out a while back, only to discover that some
>>of the
>> plugins we use were not compatible with parallel builds at the time. But
>> that may have changed.
>>
>> Our DEV@cloud builds are on micro images normally, I think (although
>>we're
>> of course not really supposed to know ;-)), but may also be running on
>>LXCs.
>> It shouldn't *hurt* to set the option, though - it just may not have any
>> effect..?
>>
>> ap
Re: Speed up jclouds builds
Posted by Ignasi Barrera <ig...@gmail.com>.
I just discovered the option and that's why I asked :)
A local build of the "jclouds" repo without and with it shows the time
you can save (this is in a machine with 4 cores):
$ mvn clean install
Total time: 08:07 min
$ mvn clean install -T1C
Total time: 03:51 min (Wall Clock)
It takes half the time. Don't know if with the instances we have we'll
see some improvement, but perhaps is worth trying. It would be cool to
reduce the PR build time.
On 28 April 2014 17:00, Andrew Phillips <ap...@qrmedia.com> wrote:
>> Has someone tried to play with the "-T" maven parameter when building
>> jclouds?
>> Do you know which kind of instances do we have in Dev@cloud, and if we
>> could set that parameter?
>
>
> I recall having tried it out a while back, only to discover that some of the
> plugins we use were not compatible with parallel builds at the time. But
> that may have changed.
>
> Our DEV@cloud builds are on micro images normally, I think (although we're
> of course not really supposed to know ;-)), but may also be running on LXCs.
> It shouldn't *hurt* to set the option, though - it just may not have any
> effect..?
>
> ap
Re: Speed up jclouds builds
Posted by Andrew Phillips <ap...@qrmedia.com>.
> Has someone tried to play with the "-T" maven parameter when
> building jclouds?
> Do you know which kind of instances do we have in Dev@cloud, and if we
> could set that parameter?
I recall having tried it out a while back, only to discover that some
of the plugins we use were not compatible with parallel builds at the
time. But that may have changed.
Our DEV@cloud builds are on micro images normally, I think (although
we're of course not really supposed to know ;-)), but may also be
running on LXCs. It shouldn't *hurt* to set the option, though - it
just may not have any effect..?
ap