[WOT] how to improve your ant build speed

[Stefano Mazzocchi <st...@apache.org>]

Pier Fumagalli wrote:
> Jason Foster wrote:
> 
>>> Sure the flowscript layer can (and will, I'm sure) be abused, but as 
>>> much as ant build files, or sitemaps can reach a point where the mess 
>>> and performance drags you down and refactoring is needed.
>>
>>
>> Since you did bring it up... ;)
>>
>> Could you offer some tips on "how to refactor an ant build script for 
>> fun and profit in order to achieve order-of-magnitude improvements"?  
>> I'm curious about the mental model you were using during the 
>> refactoring, as the details can be gleaned from the build.xml
 >
> It was a quite easy trick, to tell you the truth...

Whatever! here's the true story.

> Stefano came and visit us in January, I fiddled with his laptop screen 
> and broke it (bye bye laptop screen)...

The screen connection of my VAIO was already damaged. But it's true that 
was Pier that finally broke it. bye bye laptop screen holder... but the 
screen still works, the laptop just is not portable anymore.

> While he was trying to fix the screen he spilled some milk on the now 
> disassembled keyboard (without noticing that the glass I provided was 
> soaked in oil, and therefore very slippery).

That's BS: I got the glass myself and you were at work!

> Trying to clean the keyboard, he severed some of the contacts in the 
> membrane (whops, was there a razor-blade in that Q-Tip I gave you, Stef? 
> How unfortunate...)

You ain't gave me nothing, bro. I used TP around a screwdriver and I 
didn't know those stupid membranes were sooooooooo fragile. But now I know.

> Now, I recommend him to buy a new PowerBook-G4... To do so Stefano needs 
> to raise some money, and to do so, he can simply build a site using 
> Cocoon... 

Well, the deal was a little different: I come back home, my father needs 
better email, I say "get a provider provider", he says "why don't you do 
it", I think "doh!" and say "sure" thinking "no way". Than, frustrated 
by tons of phone calls about how shitty the whole internet is and how 
they can't do this and that (NOTE: my father is John Deere dealer. They 
sell tractors and combines. )

> So, Stefano signs the contract and orders the new PowerBook...

Stefano doesn't sign anything and starts working on the site, having his 
friend Pier hosting email and web on his machines in london (thank!) and 
we start talking about how to improve things that we don't like.

So, here comes jetty, the mod_proxy tutorial and I finally sit down and 
start.

> Ideally (how convenient), the Apple dealer is a friend of mine, and the 
> leptop shipment is misteriously delayed... OF MONTHS, I might want to 
> add...

Note that I landed in Italy a month ago. Take notes: Pier's sense of 
time is *screwed*!!! if he says "i'll be there by noon", expect him 
between 2/3PM. One time he even pretended the cops got him.

> Unfortunately Stefano has to build the site on his old home PC (_SLOW_ 
> as hell, you wouldn't believe it)...

My laptop is teared apart but still operational (well, more or less). My 
desktop was given to my father since half of his problems were due to a 
Pentium 120 with 32 mb of RAM running Win95 with IE6.0, no wonder why he 
was annoyed!

> After one week of "build run" he 
> gets frustrated and rewrites the entire build system...

After *one* run of 'build run', after seeing the "init" task passing on 
my screen for the *fifth* time, I said "that's it" and rewrote it.

> So, the whole story is, if you want someone to do something concerning 
> performance, first make sure that you are going to destroy all his 
> computers but one (the slow 6-years-old one with no RAM), you get him 
> annoyed by making him do something CPU intensive for a week, and after 
> that he'll figure out that probably it's time to rewrite stuff! :-)
> 
> Eeeeeeeeaaaaaaaaasyyyyyyyyyyy....

Pier is right in one thing: you should be building software on slower 
machines, not on those massive Ghz-based dual CPU systems.

But enough is enough and I'll be getting my Powerbook G4 next week!! Of 
somebody is going to get hurt (doesn't matter if I have to jump onto a 
plane to do it, got it buddy? :)

                                      - o -

All right, enough sunday morning stupidity.

The old build system had one serious problem: massive use of memory lead 
to massive use of fork="true" on all calls to <javac> <ant> and 
<javadoc>. Javac leaks memory, that's true, but not so much that we 
can't handle a few runs with it.

The problem with forking <ant> calls was that ant had no way to know the 
previous dependency graph, so had to re-execute all targets that were 
already previously executed.

But in order to remove the fork="true" attributes from those tasks, I 
needed to reduce memory consuption in the build.

Also, the dependency graph between targets was completely out of 
control. Tasks were dependent on tasks without reason, maybe just to 
make a directory. There was no coherent plan since it grew up from the 
work of tens of people on two years.

Also, the block-build.xml generated file had a few tasks that were 
useful only on *one* block, but where executed everytime! I still have 
to figure out a way to solve that issue, but removing that improved 
speed a lot.

Also, I removed all checks for libraries that were in blocks. I removed 
all file copying. I placed all java files in a directory structure that 
matched exactly the package name, so that ant can know not to recompile 
something.

I did *massive* care about making sure that subsequent runs went smooth, 
without requiring recompilation or repackaging.

I think there are still a few things to tweak to be faster, but it would 
be a few seconds, not more.

Anyway, the build system was simply too complex and I restarted from 
scratch and it worked. It'a painful job, expecially in a complex system 
like Cocoon's, but it's worth it. Now I can do a 'cvs update' then 
'build webapp' and takes no more than 60 seconds (less if I use jikes as 
a compiler). And this is good enough for now.

-- 
Stefano Mazzocchi                               <st...@apache.org>
    Pluralitas non est ponenda sine necessitate [William of Ockham]
--------------------------------------------------------------------