[classlib][build] Observations about build times

[Mark Hindess <ma...@googlemail.com>]

Fact: If we build all our java code twice, then a fresh build is twice
as fast and a trivial (nothing to do) build is three times faster.
(And the clean target can run 5 times faster.)

While working on cleaning up the clean target, I observed that the
"clean" in the accessibility module took much more time that the
"clean" in the x-net module.  This is a bit odd since the latter is
approximately three times larger so you'd expect if anything it would
be the other way around.

It turns out the "clever" fileset[0] used to find which of the classes
in the "big" top-level compile destination directory belong to which
module, which looks something like:

    <fileset id="classes" dir="../../build/classes">
        <or>
            <!-- only common code case -->
            <present targetdir="src/main/java" />
            <present targetdir="src/main/java">
                <mapper type="regexp"
                        from="^(.*?)(\$$[^/\\\.]*)?\.class$$"
                        to="\1.java"/>
            </present>
        </or>
    </fileset>

Unfortunately, while the patternset concisely expresses our
requirement to only taking class files for which a corresponding
source file exists this is hugely inefficient since it traverses the
very large number of files in the common ../../build/classes tree to
clean (or jar) the relatively small number of classes belonging to a
particular module.  So the reasons the clean in accessibility takes
longer is that it has to check all the classes from every module where
as x-net only has to check those that are left (i.e. those from
x-net).

Patching the build to compile every class twice - once during the big
top-level compile - and again in the module to a different location so
the classes are trivial to select (using the top-level compile on the
classpath) eliminates the need for the "clever" fileset.  Creating the
jars happens much faster.  For clean, modules delete the destdir for
their javac and the top-level just deletes the one from the top-level
compile.  The requirement to piggyback a sanity check the patternsets
during clean goes away because the patternsets are gone.

I thought this was quite interesting/amusing so I thought I'd share
it.  However, I'm really not suggesting that this is the correct
solution but I'm planning to try a few things with a view to improving
things in this area[1].

Regards,
 Mark.

[0] No prizes for guessing which idiot suggested these "clever"
filesets.  They were introduced because they were less error prone
than the manually updated patternset.txt files.  I recall the sound
module was accidentally not updated when new classes were added. See:

  http://markmail.org/thread/rjqrgudag5jxw3wt

[1] I think Alexey Petrenko might have had the right idea when he
suggested keeping the simple patternset files but automatically
generating them to avoid trivial mistakes.

Re: [classlib][build] Observations about build times

[Jesse Wilson <je...@google.com>]

On Wed, Aug 19, 2009 at 5:44 AM, Mark Hindess
<ma...@googlemail.com>wrote:

> Since the latter is rather contrived case (and extra elapsed time for me is
> only 4 seconds) then I think we can live with that.


Awesome!

Re: [classlib][build] Observations about build times

[Mark Hindess <ma...@googlemail.com>]

I've committed a change to use generated filesets in r805777.  For me,
it results in:

  a 2x speed up on a fresh build,
  a 4x speed up on a build with no changes,
  a 5x speed up on clean after a build, but
  a 0.8x speed up (i.e. slow down) on a clean after a clean

Since the latter is rather contrived case (and extra elapsed time for me is 
only 4 seconds) then I think we can live with that.

Regards,
 Mark.