ClassLoading information

[Matt Hogstrom <ma...@hogstrom.org>]

I did some profiling of DayTrader deploy (which takes almost 60  
seconds) to understand what is taking so long.  It appears that we  
spend a significant amount of time MultiParentClassLoader.loadClass 
().  We've been in here before and added some code to direct requests  
for specific class types to the SystemClassLoader (like java.,  
double, int, etc.).  This help performance a lot.  But that's old news.

I've been gathering some information on where we are spending the  
time and it appears that the single item that hurts us the most is  
classes that are not being found.  I add some code to cache the  
classes that are known to not exist in a hierarchy and the results  
were fairly impressive.

Startup time on my Mac goes from about 19 seconds to 15 seconds.   
That is just about a 20% improvement in startup time.

Deploy time for DayTrader goes from 62 seconds to 36 seconds (about  
1/2 the time).

I talked to Dain about this and its not really clear how to handle  
this.  One option is to implement the cache of classes that have not  
been found.  However, for entries in the ClassPath that are a  
directory its possible that a user might drop a class in and we would  
never find it once we failed.  We could put some kind of timer  
mechanism in to throw out entries that have not been referenced in  
some time interval.

I wanted to solicit some input on alternatives.

For those interested in how many times a loadClass fails with a CNFE  
I captured each failure by classloader and appended a number of times  
the class was looked for.  This can be found at

http://people.apache.org/~hogstrom/classNotFoundList.txt (It's about  
2MB)

or you can grab a zip file if you prefer.

http://people.apache.org/~hogstrom/classNotFoundList.zip

Re: ClassLoading information

[Aaron Mulder <am...@alumni.princeton.edu>]

One option would be climb the classloader tree to activate the cache
during startup/deployment and flush the cache when startup finishes
and when each deployment finishes.  So it would only be used during
one of those long operations and shouldn't interfere with people
dumping classes in a directory at runtime.  Granted that may be
complex to implement, but maybe less of a hassle than tracking the
last time we looked for each bad class and synchronizing all the
checks and so on.

Thanks,
       Aaron

On 4/30/07, Matt Hogstrom <ma...@hogstrom.org> wrote:
> I did some profiling of DayTrader deploy (which takes almost 60
> seconds) to understand what is taking so long.  It appears that we
> spend a significant amount of time MultiParentClassLoader.loadClass
> ().  We've been in here before and added some code to direct requests
> for specific class types to the SystemClassLoader (like java.,
> double, int, etc.).  This help performance a lot.  But that's old news.
>
> I've been gathering some information on where we are spending the
> time and it appears that the single item that hurts us the most is
> classes that are not being found.  I add some code to cache the
> classes that are known to not exist in a hierarchy and the results
> were fairly impressive.
>
> Startup time on my Mac goes from about 19 seconds to 15 seconds.
> That is just about a 20% improvement in startup time.
>
> Deploy time for DayTrader goes from 62 seconds to 36 seconds (about
> 1/2 the time).
>
> I talked to Dain about this and its not really clear how to handle
> this.  One option is to implement the cache of classes that have not
> been found.  However, for entries in the ClassPath that are a
> directory its possible that a user might drop a class in and we would
> never find it once we failed.  We could put some kind of timer
> mechanism in to throw out entries that have not been referenced in
> some time interval.
>
> I wanted to solicit some input on alternatives.
>
> For those interested in how many times a loadClass fails with a CNFE
> I captured each failure by classloader and appended a number of times
> the class was looked for.  This can be found at
>
> http://people.apache.org/~hogstrom/classNotFoundList.txt (It's about
> 2MB)
>
> or you can grab a zip file if you prefer.
>
> http://people.apache.org/~hogstrom/classNotFoundList.zip
>
>

Re: ClassLoading information

[John Sisson <jr...@gmail.com>]

Just some thoughts between nappy changes :-)

I seem to remember seeing the actual exception creation time being an 
issue when profiling the code a while ago.  I think it may be due to the 
generation of the stack trace.

It would be interesting to see how much of a performance improvement you 
would get if you reused exception objects rather than creating a new 
exception object each time a class is not found.  The problem with 
reusing exceptions is that the stack trace isn't going to be correct. 

Maybe we could have an optimization that doesn't have the stack trace or 
some partial stack information by instantiating a subclass of the 
ClassNotFoundException that overrides the fillInStackTrace() method 
(which is called by the Throwable constructor) and possibly calls the 
setStackTrace method.

You would want the ability to turn this optimization on or off.  I the 
optimization should be off by default, as it should be up to the user to 
decide whether they want to live with less information to debug the 
exception.

Regards,
John

Matt Hogstrom wrote:
> I did some profiling of DayTrader deploy (which takes almost 60 
> seconds) to understand what is taking so long.  It appears that we 
> spend a significant amount of time 
> MultiParentClassLoader.loadClass().  We've been in here before and 
> added some code to direct requests for specific class types to the 
> SystemClassLoader (like java., double, int, etc.).  This help 
> performance a lot.  But that's old news.
>
> I've been gathering some information on where we are spending the time 
> and it appears that the single item that hurts us the most is classes 
> that are not being found.  I add some code to cache the classes that 
> are known to not exist in a hierarchy and the results were fairly 
> impressive.
>
> Startup time on my Mac goes from about 19 seconds to 15 seconds.  That 
> is just about a 20% improvement in startup time.
>
> Deploy time for DayTrader goes from 62 seconds to 36 seconds (about 
> 1/2 the time).
>
> I talked to Dain about this and its not really clear how to handle 
> this.  One option is to implement the cache of classes that have not 
> been found.  However, for entries in the ClassPath that are a 
> directory its possible that a user might drop a class in and we would 
> never find it once we failed.  We could put some kind of timer 
> mechanism in to throw out entries that have not been referenced in 
> some time interval.
>
> I wanted to solicit some input on alternatives.
>
> For those interested in how many times a loadClass fails with a CNFE I 
> captured each failure by classloader and appended a number of times 
> the class was looked for.  This can be found at
>
> http://people.apache.org/~hogstrom/classNotFoundList.txt (It's about 2MB)
>
> or you can grab a zip file if you prefer.
>
> http://people.apache.org/~hogstrom/classNotFoundList.zip
>