You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Pier Fumagalli <pi...@betaversion.org> on 2004/05/25 10:49:06 UTC

VERY odd exceptions...

I'm having a bunch of problems with Avalon in production... And I  
suppose they're kinda all interconnected.

I get some odd ConcurrentModificationException(s) and  
NullPointerException(s) all coming out from the component manager  
"release/dispose" methods: I suppose that, given that the  
ConcurrentModificationException comes out from my server log, Excalibur  
catches it, and it tries to dispose of a component, but at this point  
somehow somewhere it will throw a NullPointerException (uncaught, as it  
comes out in my System.err log)...

Now, after this sequence of exceptions, from time to time, I start  
getting things like IllegalStateException(s) mentioning that I cannot  
lookup components on a disposed ComponentLocator...

And this is BAD, because to get this working again, I have to SHUT DOWN  
COCOON, clean up my sitemaps and cocoon.xconf (I mean, removing them,  
and reverting back to my CVS version, don't ask my why, I don't know),  
restart it with these new "files" (which didn't change), and the  
exception goes away...

Two problems... Definitely, under load, Excalibur fails  
disposing/releasing components, and somehow, Cocoon keeps state EVEN  
between hard restarts of the JVM...

Anyone ever saw something like this?

	Pier

java.util.ConcurrentModificationException
         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:782)
         at java.util.HashMap$KeyIterator.next(HashMap.java:818)
         at  
org.apache.avalon.excalibur.component.ExcaliburComponentManager.dispose( 
ExcaliburComponentManager.java:607)
         at  
org.apache.cocoon.components.CocoonComponentManager.dispose(CocoonCompon 
entManager.java:468)
         at  
org.apache.avalon.framework.container.ContainerUtil.dispose(ContainerUti 
l.java:345)
         at org.apache.cocoon.Cocoon.dispose(Cocoon.java:489)
         at  
org.apache.avalon.framework.container.ContainerUtil.dispose(ContainerUti 
l.java:345)
         at  
org.apache.cocoon.servlet.CocoonServlet.disposeCocoon(CocoonServlet.java 
:1463)
         at  
org.apache.cocoon.servlet.CocoonServlet.destroy(CocoonServlet.java:497)
         at  
org.mortbay.jetty.servlet.ServletHolder.stop(ServletHolder.java:244)
         at  
org.mortbay.jetty.servlet.ServletHandler.stop(ServletHandler.java:470)
         at  
org.mortbay.jetty.servlet.WebApplicationHandler.stop(WebApplicationHandl 
er.java:163)
         at org.mortbay.http.HttpContext.stop(HttpContext.java:1979)
         at  
org.mortbay.jetty.servlet.ServletHttpContext.stop(ServletHttpContext.jav 
a:145)
         at  
org.mortbay.jetty.servlet.WebApplicationContext.stop(WebApplicationConte 
xt.java:504)
         at org.mortbay.http.HttpContext.stop(HttpContext.java:1950)
         at org.mortbay.http.HttpServer.stop(HttpServer.java:732)
         at org.mortbay.http.HttpServer.stop(HttpServer.java:700)
         at org.mortbay.jetty.Server$1.run(Server.java:455)

java.lang.NullPointerException
         at  
org.apache.cocoon.components.CocoonComponentManager.release(CocoonCompon 
entManager.java:507)
         at  
org.apache.cocoon.environment.AbstractEnvironment.release(AbstractEnviro 
nment.java:539)
         at  
org.apache.cocoon.environment.wrapper.MutableEnvironmentFacade.release(M 
utableEnvironmentFacade.java:320)
         at  
org.apache.cocoon.transformation.TraxTransformer.recycle(TraxTransformer 
.java:514)
         at  
org.apache.avalon.excalibur.pool.ResourceLimitingPool.put(ResourceLimiti 
ngPool.java:438)
         at  
org.apache.avalon.excalibur.component.PoolableComponentHandler.doPut(Poo 
lableComponentHandler.java:212)
         at  
org.apache.avalon.excalibur.component.ComponentHandler.put(ComponentHand 
ler.java:425)
         at  
org.apache.avalon.excalibur.component.ExcaliburComponentSelector.release 
(ExcaliburComponentSelector.java:305)
         at  
org.apache.cocoon.components.ExtendedComponentSelector.release(ExtendedC 
omponentSelector.java:284)
         at  
org.apache.cocoon.components.ExtendedComponentSelector.release(ExtendedC 
omponentSelector.java:281)
         at  
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.recycle 
(AbstractProcessingPipeline.java:652)
         at  
org.apache.avalon.excalibur.pool.ResourceLimitingPool.put(ResourceLimiti 
ngPool.java:438)
         at  
org.apache.avalon.excalibur.component.PoolableComponentHandler.doPut(Poo 
lableComponentHandler.java:212)
         at  
org.apache.avalon.excalibur.component.ComponentHandler.put(ComponentHand 
ler.java:425)
         at  
org.apache.avalon.excalibur.component.ExcaliburComponentSelector.release 
(ExcaliburComponentSelector.java:305)
         at  
org.apache.cocoon.components.ExtendedComponentSelector.release(ExtendedC 
omponentSelector.java:284)
         at  
org.apache.cocoon.components.EnvironmentDescription.removeFromAutoReleas 
e(CocoonComponentManager.java:705)
         at  
org.apache.cocoon.components.CocoonComponentManager.removeFromAutomaticR 
elease(CocoonComponentManager.java:445)
         at  
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.release 
(AbstractProcessingPipeline.java:173)
         at  
org.apache.cocoon.components.source.impl.SitemapSource.reset(SitemapSour 
ce.java:454)
         at  
org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSour 
ce.java:445)
         at  
org.apache.cocoon.components.source.SourceUtil.parse(SourceUtil.java: 
247)
         at  
org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java: 
106)
         at  
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process 
XMLPipeline(AbstractProcessingPipeline.java:546)
         at  
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process 
(AbstractProcessingPipeline.java:490)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke( 
SerializeNode.java:120)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:49)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.i 
nvoke(PreparableMatchNode.java:130)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(P 
ipelineNode.java:126)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke( 
PipelinesNode.java:101)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:336)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:277)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(Moun 
tNode.java:103)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:49)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.i 
nvoke(PreparableMatchNode.java:130)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(P 
ipelineNode.java:126)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke( 
PipelinesNode.java:101)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:336)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:277)
         at org.apache.cocoon.Cocoon.process(Cocoon.java:619)
         at  
org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java: 
1074)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at  
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:354)
         at  
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationH 
andler.java:294)
         at  
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 
567)
         at org.mortbay.http.HttpContext.handle(HttpContext.java:1808)
         at  
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationCon 
text.java:525)
         at org.mortbay.http.HttpContext.handle(HttpContext.java:1758)
         at org.mortbay.http.HttpServer.service(HttpServer.java:879)
         at  
org.mortbay.http.HttpConnection.service(HttpConnection.java:790)
         at  
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:952)
         at  
org.mortbay.http.HttpConnection.handle(HttpConnection.java:807)
         at  
org.mortbay.http.SocketListener.handleConnection(SocketListener.java: 
197)
         at  
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
         at  
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:501)

java.lang.IllegalStateException: You cannot lookup components on a  
disposed ComponentLocator
         at  
org.apache.avalon.excalibur.component.ExcaliburComponentManager.lookup(E 
xcaliburComponentManager.java:167)
         at  
org.apache.cocoon.components.CocoonComponentManager.lookup(CocoonCompone 
ntManager.java:297)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.ErrorHandlerHelper.in 
vokeErrorHandler(ErrorHandlerHelper.java:65)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(P 
ipelineNode.java:149)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke( 
PipelinesNode.java:101)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:336)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:277)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(Moun 
tNode.java:103)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:49)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.i 
nvoke(PreparableMatchNode.java:130)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(P 
ipelineNode.java:126)
         at  
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode. 
invokeNodes(AbstractParentProcessingNode.java:72)
         at  
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke( 
PipelinesNode.java:101)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:336)
         at  
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreePro 
cessor.java:277)
         at org.apache.cocoon.Cocoon.process(Cocoon.java:619)
         at  
org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java: 
1074)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at  
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:354)
         at  
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationH 
andler.java:294)
         at  
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 
567)
         at org.mortbay.http.HttpContext.handle(HttpContext.java:1808)
         at  
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationCon 
text.java:525)
         at org.mortbay.http.HttpContext.handle(HttpContext.java:1758)
         at org.mortbay.http.HttpServer.service(HttpServer.java:879)
         at  
org.mortbay.http.HttpConnection.service(HttpConnection.java:790)
         at  
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:952)
         at  
org.mortbay.http.HttpConnection.handle(HttpConnection.java:807)
         at  
org.mortbay.http.SocketListener.handleConnection(SocketListener.java: 
197)
         at  
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
         at  
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:501)

Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 10:34, Bertrand Delacretaz wrote:
> Le 25 mai 04, à 11:31, Pier Fumagalli a écrit :
>
>> ...Have you noticed, by any chance, a ConcurrentModificationException 
>> somewhere in your tests?
>
> Not that I remember, but I've attached some logs in bugzilla, you 
> might want to dig in them.

Nothing in your logs... But it must have been on your console or in 
your standard error/output or servlet container log... It doesn't get 
logged by Cocoon...

>>> IIRC, hard restarts didn't help either in this case, the only "fix" 
>>> was to edit the sitemap (add a space somewhere for example).
>>
>> How come we're keeping state on the sitemap? Where?
>
> No idea, really. But the CVS commit scenario might be the easiest way 
> if you want to reproduce it.

Oh, if you spend more than 5 minutes on www.vnunet.com you'll reproduce 
it yourself! :-) For me, it's just a matter of tailing the log files! 
:-P

	Pier


Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 25 mai 04, à 11:31, Pier Fumagalli a écrit :

> ...Have you noticed, by any chance, a ConcurrentModificationException 
> somewhere in your tests?

Not that I remember, but I've attached some logs in bugzilla, you might 
want to dig in them.

>
>> IIRC, hard restarts didn't help either in this case, the only "fix" 
>> was to edit the sitemap (add a space somewhere for example).
>
> How come we're keeping state on the sitemap? Where?

No idea, really. But the CVS commit scenario might be the easiest way 
if you want to reproduce it.

-Bertrand


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 13:40, Berin Loritsch wrote:
> Pier Fumagalli wrote:
>> On 25 May 2004, at 10:05, Bertrand Delacretaz wrote:
>> Have you noticed, by any chance, a ConcurrentModificationException 
>> somewhere in your tests?
>> I noticed that the ExcaliburComponentManager doesn't correctly 
>> synchronize on its m_componentHandlers hashmap, and that might be the 
>> root of our problems...
>
> A ConcurrentModificationException happens when you invoke a "put" or
> "remove" operation on the HashMap at the same time you are using an
> Iterator.  The Iterator will throw the ConcurrentModificationException
> because the information that the iterator is going through is no longer
> valid.
>
> I don't recall there being anything in the ECM dispose method that 
> would
> cause this to happen.  There are a couple things I would check though:
>
> * Request scoped lookups
> * The BucketMap in Commons Collections
>
> I would double check ECM as well--I just have to get another release
> done at work before noon.  After that, I will see what I can dig up.

I just posted a patch to BugZilla which fixes the 
ConcurrentModificationException I am observing. Per each page I look up 
few bazillion components (roughly a couple of hundred AFAICS) in few 
bazillion different sitemaps (four).

	Pier

Re: VERY odd exceptions...

Posted by Berin Loritsch <bl...@d-haven.org>.
Pier Fumagalli wrote:
> On 25 May 2004, at 10:05, Bertrand Delacretaz wrote:
> 
> Have you noticed, by any chance, a ConcurrentModificationException 
> somewhere in your tests?
> 
> I noticed that the ExcaliburComponentManager doesn't correctly 
> synchronize on its m_componentHandlers hashmap, and that might be the 
> root of our problems...

A ConcurrentModificationException happens when you invoke a "put" or
"remove" operation on the HashMap at the same time you are using an
Iterator.  The Iterator will throw the ConcurrentModificationException
because the information that the iterator is going through is no longer
valid.

I don't recall there being anything in the ECM dispose method that would
cause this to happen.  There are a couple things I would check though:

* Request scoped lookups
* The BucketMap in Commons Collections

I would double check ECM as well--I just have to get another release
done at work before noon.  After that, I will see what I can dig up.

> 
>> IIRC, hard restarts didn't help either in this case, the only "fix" 
>> was to edit the sitemap (add a space somewhere for example).
> 
> 
> How come we're keeping state on the sitemap? Where?

When the sitemap was compiled that was an easy one.  Perhaps the "store"
or cache is the place where that state is maintained?


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 10:05, Bertrand Delacretaz wrote:
> Le 25 mai 04, à 10:49, Pier Fumagalli a écrit :
>> ...Two problems... Definitely, under load, Excalibur fails 
>> disposing/releasing components, and somehow, Cocoon keeps state EVEN 
>> between hard restarts of the JVM...
>>
>> Anyone ever saw something like this?
>
> Might be related to this very mysterious bug
> http://nagoya.apache.org/bugzilla/show_bug.cgi?id=27249
> which I've seen from time to time in all versions since 
> cocoon-2.1rc2-dev (at least).
>
> My comment of 2004-02-27 10:23 in the bug gives a reproducible case.

Yep, seems to be the same bug... And it's happening _A_LOT_ on our 
production environment :-( :-(

Have you noticed, by any chance, a ConcurrentModificationException 
somewhere in your tests?

I noticed that the ExcaliburComponentManager doesn't correctly 
synchronize on its m_componentHandlers hashmap, and that might be the 
root of our problems...

> IIRC, hard restarts didn't help either in this case, the only "fix" 
> was to edit the sitemap (add a space somewhere for example).

How come we're keeping state on the sitemap? Where?

	Pier

Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 25 mai 04, à 10:49, Pier Fumagalli a écrit :
> ...Two problems... Definitely, under load, Excalibur fails 
> disposing/releasing components, and somehow, Cocoon keeps state EVEN 
> between hard restarts of the JVM...
>
> Anyone ever saw something like this?

Might be related to this very mysterious bug
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=27249
which I've seen from time to time in all versions since 
cocoon-2.1rc2-dev (at least).

My comment of 2004-02-27 10:23 in the bug gives a reproducible case.

IIRC, hard restarts didn't help either in this case, the only "fix" was 
to edit the sitemap (add a space somewhere for example).

-Bertrand


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 10:27, Bertrand Delacretaz wrote:

> Le 25 mai 04, à 11:17, Sylvain Wallez a écrit :
>
>>
>> ...But what you describe here is really weird: there's absolutely no 
>> data related to sitemap or component manager that is persisted to 
>> disk.
>
> Although in bug 27249 I've seen the problem persist after restarting 
> cocoon *and* deleting the Jetty temp directory. Weird de chez weird.

I'd call this "spooky"...

	Pier



Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 25 mai 04, à 11:17, Sylvain Wallez a écrit :

>
> ...But what you describe here is really weird: there's absolutely no 
> data related to sitemap or component manager that is persisted to 
> disk.

Although in bug 27249 I've seen the problem persist after restarting 
cocoon *and* deleting the Jetty temp directory. Weird de chez weird.

-Bertrand


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 10:17, Sylvain Wallez wrote:
> Pier Fumagalli wrote:
>
>> I'm having a bunch of problems with Avalon in production... And I  
>> suppose they're kinda all interconnected.
>>
>> I get some odd ConcurrentModificationException(s) and  
>> NullPointerException(s) all coming out from the component manager  
>> "release/dispose" methods: I suppose that, given that the  
>> ConcurrentModificationException comes out from my server log, 
>> Excalibur  catches it, and it tries to dispose of a component, but at 
>> this point  somehow somewhere it will throw a NullPointerException 
>> (uncaught, as it  comes out in my System.err log)...
>>
>> Now, after this sequence of exceptions, from time to time, I start  
>> getting things like IllegalStateException(s) mentioning that I cannot 
>>  lookup components on a disposed ComponentLocator...
>>
>> And this is BAD, because to get this working again, I have to SHUT 
>> DOWN  COCOON, clean up my sitemaps and cocoon.xconf (I mean, removing 
>> them,  and reverting back to my CVS version, don't ask my why, I 
>> don't know),  restart it with these new "files" (which didn't 
>> change), and the  exception goes away...
>
>
> Bertrand's problem related to CVS $Id$ in comments makes me think to 
> clock synchronisation problems between the CVS server and the Cocoon 
> server (e.g. modification date moved back in time).

If the modification time is moved back in time (for example), I would 
expect a behavior of NOT reloading the sitemap... Not of throwing an 
exception :-(

> But what you describe here is really weird: there's absolutely no data 
> related to sitemap or component manager that is persisted to disk.

Bertrand has the same exact problem...

> Also, is your problem occuring when nothing changes in the system, or 
> is it related to changes in the sitemap, like Bertrand's case?

Site's running on production... NOTHING changes... Noone but me has 
access to those machines, furthermore, the application is checked out 
from SubVersion, and "svn status" doesn't report any whatsoever change.

	Pier

Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 26 mai 04, à 10:56, Ugo Cei a écrit :

> ..."The Java language specification allows access and storage of 
> [variables of type double and long] (but not those of other scalar 
> types) to be non-atomic."...

Hmm..smells too much like something that could be interpreted 
differently by different VMs.
Me, I'd stay on the safe side and synchronize.

-Bertrand, better safe with a few millipercent performance lost than 
sorry ;-)


Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Joerg Heinicke wrote:

> On 26.05.2004 10:56, Ugo Cei wrote:
>
>> Sylvain Wallez wrote:
>>
>>> I was just looking in the VM spec if a increment operation on an 
>>> integer is atomic or not. My opinion it has to be in the case of 
>>> class members as the increment is surrounded by getfield/putfield 
>>> instructions.
>>
>>
>>
>> Accessing an "int" is atomic. Accessing a "long" is not. This is all 
>> from memory, which might fail, of course ... so let me see if I can 
>> dig up a reference...
>
>
> But an increment is more than an access, isn't it?


It's three operations: read access, add one, write access. Thus, it is 
not atomic. Hence, synchronisation is required. That's the first 
question you ask when interviewing someone - what would unsynchronized 
counter in servlet show? And in EJB? :-)

Vadim



Re: VERY odd exceptions...

Posted by Joerg Heinicke <jo...@gmx.de>.
On 26.05.2004 10:56, Ugo Cei wrote:
> Sylvain Wallez wrote:
> 
>> I was just looking in the VM spec if a increment operation on an 
>> integer is atomic or not. My opinion it has to be in the case of class 
>> members as the increment is surrounded by getfield/putfield instructions.
> 
> 
> Accessing an "int" is atomic. Accessing a "long" is not. This is all 
> from memory, which might fail, of course ... so let me see if I can dig 
> up a reference...

But an increment is more than an access, isn't it?

Joerg

Re: VERY odd exceptions...

Posted by Ugo Cei <u....@cbim.it>.
Ugo Cei wrote:
> I don't think it is necessary, and could hurt performance, but before I 
> try to dig up some other reference, I'd like to know if we're talking 
> about the same scenario.

OK, turns out my memory is getting worse each day. I was wrong, 
incrementing an "int" is most certainly *NOT* an atomic operation.

References:

J. Bloch, "Effective Java", page 190.
B. Eckel, "Thinking in Java, 2nd Ed.", page 734.

I am glad that I have a large selection of technical books sitting on my 
desk to complement my failing memory ;-)

	Ugo


Re: VERY odd exceptions...

Posted by Berin Loritsch <bl...@d-haven.org>.
Carsten Ziegeler wrote:

> Yes and I think we already went through this hell as we introduced
> the concurrent request counter in Cocoon.java which is now only
> enabled if debug logging is enabled.
> 
> In our case this counter is used for disposing a sitemap that has
> been changed. Now we could argue that this might not be a feature
> that is used in production...
> 
> At least, if it creates a bottleneck it should somehow be possible
> to turn this extra syncing off.

Hmm. So you are referring to reference counting as the major mechanism 
here?  :(

Since you already have the Doug Lea Concurrent library (did you upgrade 
to the newest version?) why not simply use a FutureResult?

I did this recently for a database bound Swing app, and the difference 
in perceived performance is incredible.  In order to do this properly,
you would have something like this:

public class MyClass implements Initializable {
	FutureResult m_sitemap = new FutureResult();
	Executor m_executor = // ...;
	// ...

	public void initialize() {
		m_executor.execute(
			m_sitemap.setter(new SitemapCallable());
	}

	public void reload() {
		// note: if you want to make all threads wait for
		// us to set up the new sitemap instead of continue
		// to use the old one in the mean time, comment out
		// the following line:
		// m_sitemap.clear();
		m_executor.execute(
			m_sitemap.setter(new SitemapCallable());
	}

	public Sitemap getSitemap() {
		Sitemap map = null;

		try {
			m_sitemap.get();
		} catch (Exception e) {
			// handle it
		}

		return map;
	}
}

public class SitemapCallable implements Callable {
	public Object call() throws Exception {
		// set up the sitemap
	}
}

This is much more reliable than reference counting.  Also note that you
could potentially just reload the sitemap every so often without regard
to the last modified date and all will continue to be ok and operate
rather smoothly.


Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Leo Sutic wrote:

<snip/>

>Of course, when the sitemap is up for reloading, we can't accept any
>new requests (otherwise you get starvation). Perhaps putting a
>read-write lock around the treeprocessor would be better and easier?
>  
>

The sitemap reloading is in a synchronized block, meaning that requests 
coming while the sitemap is being read are suspendend until the 
processor is built.

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Leo Sutic <le...@gmail.com>.
On Wed, 26 May 2004 09:24:50 -0400, Berin Loritsch
<bl...@d-haven.org> wrote:
> The more correct version would be:
> 
> public class MyClass {
>     private volatile int counter; //atomic
> 
>     public someMethod() {
>         ++counter;
>     }
> }
> 
> Only longs on up should be actually synchronized.

No, because a ++ is still a load-add-store, volatile only makes this
equivalent to:

int temp = 0;
synchronized (lock) {
    temp = counter;
}
temp = temp + 1;
synchronized (lock) {
    counter = temp;
}

Of course some other thread can jump in between the synchronized
blocks and modify the counter variable.

longs should be synchronized because they are written in two
instructions, meaning that while you may get a stale value when
accessing an unsynchronized int, you might just get a jumble of bits
when reading an unsync long (you may read id *while the write is in
progress*).

> I agree that either approach *will* without a doubt cause a bottleneck
> and hurt the scalability of Cocoon.

I disagree. I'm sure the time spent synchronizing on two blocks as
small as that will not have any impact that can be measured.

Of course, when the sitemap is up for reloading, we can't accept any
new requests (otherwise you get starvation). Perhaps putting a
read-write lock around the treeprocessor would be better and easier?

/LS

RE: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
Berin Loritsch wrote:
> 
> Ugo Cei wrote:
> 
> > Sylvain Wallez wrote:
> > 
> >> Agree for *variables*. Here, we have a class member, which 
> has to be 
> >> loaded, then incremented, then stored.
> >>
> >> So as Bertrand, I stay on the safe side and synchronize.
> > 
> > 
> > Just to be clear, what you're suggesting is to do:
> > 
> > public class MyClass {
> > 
> >   private int counter;
> > 
> >   public someMethod() {
> >     ...
> >     synchronized(this) {
> >       ++counter;
> >     }
> >     ...
> >   }
> >   ...
> > }
> > 
> > ?
> > 
> > I don't think it is necessary, and could hurt performance, 
> but before 
> > I try to dig up some other reference, I'd like to know if we're 
> > talking about the same scenario.
> > 
> >     Ugo
> 
> 
> The more correct version would be:
> 
> public class MyClass {
>      private volatile int counter; //atomic
> 
>      public someMethod() {
>          ++counter;
>      }
> }
> 
> 
> Only longs on up should be actually synchronized.
> 
> I agree that either approach *will* without a doubt cause a 
> bottleneck and hurt the scalability of Cocoon.
> 
Yes and I think we already went through this hell as we introduced
the concurrent request counter in Cocoon.java which is now only
enabled if debug logging is enabled.

In our case this counter is used for disposing a sitemap that has
been changed. Now we could argue that this might not be a feature
that is used in production...

At least, if it creates a bottleneck it should somehow be possible
to turn this extra syncing off.

Carsten 


Re: VERY odd exceptions...

Posted by Berin Loritsch <bl...@d-haven.org>.
Ugo Cei wrote:

> Sylvain Wallez wrote:
> 
>> Agree for *variables*. Here, we have a class member, which has to be 
>> loaded, then incremented, then stored.
>>
>> So as Bertrand, I stay on the safe side and synchronize.
> 
> 
> Just to be clear, what you're suggesting is to do:
> 
> public class MyClass {
> 
>   private int counter;
> 
>   public someMethod() {
>     ...
>     synchronized(this) {
>       ++counter;
>     }
>     ...
>   }
>   ...
> }
> 
> ?
> 
> I don't think it is necessary, and could hurt performance, but before I 
> try to dig up some other reference, I'd like to know if we're talking 
> about the same scenario.
> 
>     Ugo


The more correct version would be:

public class MyClass {
     private volatile int counter; //atomic

     public someMethod() {
         ++counter;
     }
}


Only longs on up should be actually synchronized.

I agree that either approach *will* without a doubt cause a bottleneck 
and hurt the scalability of Cocoon.


Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Ugo Cei wrote:

> Sylvain Wallez wrote:
>
>> Agree for *variables*. Here, we have a class member, which has to be 
>> loaded, then incremented, then stored.
>>
>> So as Bertrand, I stay on the safe side and synchronize.
>
>
> Just to be clear, what you're suggesting is to do:
>
> public class MyClass {
>
>   private int counter;
>
>   public someMethod() {
>     ...
>     synchronized(this) {
>       ++counter;
>     }
>     ...
>   }
>   ...
> }


That's exactly this! More precisely:

public someMethod() {

    synchronized(this) {
        counter++;
    }
    try {
        blah();
    } finally {
        synchronized(this) {
            counter--;
        }
}

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Ugo Cei <u....@cbim.it>.
Sylvain Wallez wrote:
> Agree for *variables*. Here, we have a class member, which has to be 
> loaded, then incremented, then stored.
> 
> So as Bertrand, I stay on the safe side and synchronize.

Just to be clear, what you're suggesting is to do:

public class MyClass {

   private int counter;

   public someMethod() {
     ...
     synchronized(this) {
       ++counter;
     }
     ...
   }
   ...
}

?

I don't think it is necessary, and could hurt performance, but before I 
try to dig up some other reference, I'd like to know if we're talking 
about the same scenario.

	Ugo



Synchronization (was Re: VERY odd exceptions...)

Posted by Berin Loritsch <bl...@d-haven.org>.
Sylvain Wallez wrote:
> 
> Agree for *variables*. Here, we have a class member, which has to be 
> loaded, then incremented, then stored.
> 
> So as Bertrand, I stay on the safe side and synchronize.
> 
> Sylvain

I understand the desire for more metrics and instrumenting Cocoon so 
that you can get the information you desire, but I have a very serious 
caveat emptor here.  The more synchronization you put into the system 
the less scalability you get.  It comes at a very heavy price.  See my 
blog for a word picture of thread contention:

http://www.jroller.com/comments/bloritsch/Weblog/fighting_over_resources

Keep in mind that the more order you try and impose on a chaotic 
atmosphere the more bottlenecks you introduce.  The contention becomes 
more violent and the overall system slows down.

If you go this route, I highly recommend you place it inside a block 
that does not execute if debugging is not happening.  For example:

if ( getLogger().isDebugEnabled() ) {
	// do synchronized instrumentation
}

That way in production you do not have to pay the price later.


Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Ugo Cei wrote:

> Sylvain Wallez wrote:
>
>> I was just looking in the VM spec if a increment operation on an 
>> integer is atomic or not. My opinion it has to be in the case of 
>> class members as the increment is surrounded by getfield/putfield 
>> instructions.
>
>
> Accessing an "int" is atomic. Accessing a "long" is not. This is all 
> from memory, which might fail, of course ... so let me see if I can 
> dig up a reference...
>
> There! Doug Lea, "Concurrent Programming in Java", p. 63:
>
> "The Java language specification allows access and storage of 
> [variables of type double and long] (but not those of other scalar 
> types) to be non-atomic."


Agree for *variables*. Here, we have a class member, which has to be 
loaded, then incremented, then stored.

So as Bertrand, I stay on the safe side and synchronize.

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Ugo Cei <u....@cbim.it>.
Sylvain Wallez wrote:
> I was just looking in the VM spec if a increment operation on an integer 
> is atomic or not. My opinion it has to be in the case of class members 
> as the increment is surrounded by getfield/putfield instructions.

Accessing an "int" is atomic. Accessing a "long" is not. This is all 
from memory, which might fail, of course ... so let me see if I can dig 
up a reference...

There! Doug Lea, "Concurrent Programming in Java", p. 63:

"The Java language specification allows access and storage of [variables 
of type double and long] (but not those of other scalar types) to be 
non-atomic."

	Ugo



Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Carsten Ziegeler wrote:

>Sylvain Wallez wrote:
>  
>
>>Spotted the problems, which are actually twofold in the TreeProcessor:
>>- sitemap modification date is checked to be before or equal 
>>to the date when the treeprocessor was built, meaning 
>>continuous rebuild if the sitemap date is in the future
>>- treeprocessor disposal (and hence its component manager) 
>>occurs as soon as change is detected, even if some other 
>>request (either in another thread or through a "cocoon:") 
>>currently use that treeprocessor, leading to the 
>>IllegalStateExceptions
>>
>>The first problem is easy to solve, but the second one is 
>>more tricky. 
>>My current thinking about it is to add a concurrent requests 
>>counter and disposal flag. When a sitemap change is detected 
>>the flag is set to true. The treeprocessor will be disposed 
>>if the disposal flag is true _and_ the concurrent requests 
>>counter is zero.
>>
>>    
>>
>I think we have the second problem in Cocoon since our first version :(
>I always thought about adding a counter for the concurrent requests
>in a tree processor, but always forget about it :( Regarding
>performance: is this a problem as the counter has to be synchronized?
>  
>

I was just looking in the VM spec if a increment operation on an integer 
is atomic or not. My opinion it has to be in the case of class members 
as the increment is surrounded by getfield/putfield instructions.

But the overhead should be minimal, as only the increment/decrement 
operation have to be synchronized.

Sylvain, currently working on the fix.

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


RE: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
 
Ugo Cei wrote:
> 
> Carsten Ziegeler wrote:
> > I think we have the second problem in Cocoon since our 
> first version 
> > :( I always thought about adding a counter for the 
> concurrent requests 
> > in a tree processor, but always forget about it :( Regarding
> > performance: is this a problem as the counter has to be 
> synchronized?
> 
> Whay? If the counter is a simple "int" field, you don't need 
> to synchronize access to it, or I am missing something?
> 
Sure, but it's somehow synchronized of course internally. Now I 
think to remember a problem in Cocoon.java where we counted
all active requests. I think we changed this because of performance
to only count if debug logging is turned on. That's why I 
thought about this problem. But perhaps I suffer from a weak memory :)

Carsten


Re: VERY odd exceptions...

Posted by Ugo Cei <u....@cbim.it>.
Carsten Ziegeler wrote:
> I think we have the second problem in Cocoon since our first version :(
> I always thought about adding a counter for the concurrent requests
> in a tree processor, but always forget about it :( Regarding
> performance: is this a problem as the counter has to be synchronized?

Whay? If the counter is a simple "int" field, you don't need to 
synchronize access to it, or I am missing something?

	Ugo



RE: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
Sylvain Wallez wrote:
> 
> Spotted the problems, which are actually twofold in the TreeProcessor:
> - sitemap modification date is checked to be before or equal 
> to the date when the treeprocessor was built, meaning 
> continuous rebuild if the sitemap date is in the future
> - treeprocessor disposal (and hence its component manager) 
> occurs as soon as change is detected, even if some other 
> request (either in another thread or through a "cocoon:") 
> currently use that treeprocessor, leading to the 
> IllegalStateExceptions
> 
> The first problem is easy to solve, but the second one is 
> more tricky. 
> My current thinking about it is to add a concurrent requests 
> counter and disposal flag. When a sitemap change is detected 
> the flag is set to true. The treeprocessor will be disposed 
> if the disposal flag is true _and_ the concurrent requests 
> counter is zero.
> 
I think we have the second problem in Cocoon since our first version :(
I always thought about adding a counter for the concurrent requests
in a tree processor, but always forget about it :( Regarding
performance: is this a problem as the counter has to be synchronized?

Carsten


RE: Start 2.1.6, Re: [status update] Re: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
 

> -----Original Message-----
> From: Vadim Gritsenko [mailto:vadim@reverycodes.com] 
> Sent: Wednesday, May 26, 2004 2:30 PM
> To: dev@cocoon.apache.org
> Subject: Start 2.1.6, Re: [status update] Re: VERY odd exceptions...
> 
> Carsten Ziegeler wrote:
> 
> >Great!
> >
> >As our head contains 2.2-dev already, I guess we should 
> start thinking 
> >about branching for 2.1.6, right?
> >  
> >
> 
> Yep. Can you create a branch?
> 
Sure. The question is: do we want to create the branch now or do we
wait until the svn move?

Carsten


Start 2.1.6, Re: [status update] Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Carsten Ziegeler wrote:

>Great!
>
>As our head contains 2.2-dev already, I guess we should start thinking
>about branching for 2.1.6, right?
>  
>

Yep. Can you create a branch?

Vadim


RE: [status update] Re: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
Sylvain Wallez wrote:
> 
> Carsten Ziegeler wrote:
> 
> >Great!
> >
> >As our head contains 2.2-dev already, I guess we should 
> start thinking 
> >about branching for 2.1.6, right?
> >  
> >
> 
> Doh! I certainly missed something (was really busy when the 
> move to SVN was discussed and skipped that thread). Is the 
> "cocoon-2.1" repository holding the 2.2-dev? 
Yes.

> If so what's in 
> the "cocoon-2.2" repo?
> 
Good question :) The cocoon-2.2 repository can be currently seen
as our "sandbox". It has two additional things in it: the try
to move to Fortress and Pier's code for the block system.
Neither of the two should block our development efforts. So,
we simply continue with the "usual" development of Cocoon (2.2.x)
and if some new "big things" like blocks etc. are mature they
will be "merged" into the "usual" development road.

When we finally move to subversion we can clean this up.

Carsten


Re: [status update] Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Carsten Ziegeler wrote:

>Great!
>
>As our head contains 2.2-dev already, I guess we should start thinking
>about branching for 2.1.6, right?
>  
>

Doh! I certainly missed something (was really busy when the move to SVN 
was discussed and skipped that thread). Is the "cocoon-2.1" repository 
holding the 2.2-dev? If so what's in the "cocoon-2.2" repo?

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


RE: [status update] Re: VERY odd exceptions...

Posted by Carsten Ziegeler <cz...@s-und-n.de>.
Great!

As our head contains 2.2-dev already, I guess we should start thinking
about branching for 2.1.6, right?

Carsten 

> -----Original Message-----
> From: Sylvain Wallez [mailto:sylvain@apache.org] 
> Sent: Wednesday, May 26, 2004 1:56 PM
> To: dev@cocoon.apache.org
> Subject: [status update] Re: VERY odd exceptions...
> 
> Sylvain Wallez wrote:
> 
> > Bertrand Delacretaz wrote:
> >
> >>>   touch 12011200 sitemap.xmap
> >>>
> >>> reproduces the disposed ComponentLocator problem!
> >>
> >>
> >>
> >> Definitely reproducible, see new comments in
> >> http://issues.apache.org/bugzilla/show_bug.cgi?id=27249
> >
> >
> >
> > Spotted the problems, which are actually twofold in the 
> TreeProcessor:
> > - sitemap modification date is checked to be before or equal to the 
> > date when the treeprocessor was built, meaning continuous 
> rebuild if 
> > the sitemap date is in the future
> > - treeprocessor disposal (and hence its component manager) 
> occurs as 
> > soon as change is detected, even if some other request (either in 
> > another thread or through a "cocoon:") currently use that 
> > treeprocessor, leading to the IllegalStateExceptions
> >
> > The first problem is easy to solve, but the second one is 
> more tricky. 
> > My current thinking about it is to add a concurrent 
> requests counter 
> > and disposal flag. When a sitemap change is detected the 
> flag is set 
> > to true. The treeprocessor will be disposed if the disposal flag is 
> > true _and_ the concurrent requests counter is zero.
> >
> > Time to sleep now. I'll see if this still seems to be a 
> good solution 
> > when waking up and implement it, unless someone else is 
> faster or has 
> > a better idea ;-)
> 
> 
> The fix is nearly ready on my Mac. It required some 
> refactoring of the TreeProcessor, now split into two classes. 
> First tests seem to run fine, but I'll be out at a customer 
> this afternoon and will be able to commit it only tomorrow morning.
> 
> Sylvain
> 
> -- 
> Sylvain Wallez                                  Anyware Technologies
> http://www.apache.org/~sylvain           http://www.anyware-tech.com
> { XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }
> 
> 


[status update] Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Sylvain Wallez wrote:

> Bertrand Delacretaz wrote:
>
>>>   touch 12011200 sitemap.xmap
>>>
>>> reproduces the disposed ComponentLocator problem!
>>
>>
>>
>> Definitely reproducible, see new comments in
>> http://issues.apache.org/bugzilla/show_bug.cgi?id=27249
>
>
>
> Spotted the problems, which are actually twofold in the TreeProcessor:
> - sitemap modification date is checked to be before or equal to the 
> date when the treeprocessor was built, meaning continuous rebuild if 
> the sitemap date is in the future
> - treeprocessor disposal (and hence its component manager) occurs as 
> soon as change is detected, even if some other request (either in 
> another thread or through a "cocoon:") currently use that 
> treeprocessor, leading to the IllegalStateExceptions
>
> The first problem is easy to solve, but the second one is more tricky. 
> My current thinking about it is to add a concurrent requests counter 
> and disposal flag. When a sitemap change is detected the flag is set 
> to true. The treeprocessor will be disposed if the disposal flag is 
> true _and_ the concurrent requests counter is zero.
>
> Time to sleep now. I'll see if this still seems to be a good solution 
> when waking up and implement it, unless someone else is faster or has 
> a better idea ;-)


The fix is nearly ready on my Mac. It required some refactoring of the 
TreeProcessor, now split into two classes. First tests seem to run fine, 
but I'll be out at a customer this afternoon and will be able to commit 
it only tomorrow morning.

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Bertrand Delacretaz wrote:

>>   touch 12011200 sitemap.xmap
>>
>> reproduces the disposed ComponentLocator problem!
>
>
> Definitely reproducible, see new comments in
> http://issues.apache.org/bugzilla/show_bug.cgi?id=27249


Spotted the problems, which are actually twofold in the TreeProcessor:
- sitemap modification date is checked to be before or equal to the date 
when the treeprocessor was built, meaning continuous rebuild if the 
sitemap date is in the future
- treeprocessor disposal (and hence its component manager) occurs as 
soon as change is detected, even if some other request (either in 
another thread or through a "cocoon:") currently use that treeprocessor, 
leading to the IllegalStateExceptions

The first problem is easy to solve, but the second one is more tricky. 
My current thinking about it is to add a concurrent requests counter and 
disposal flag. When a sitemap change is detected the flag is set to 
true. The treeprocessor will be disposed if the disposal flag is true 
_and_ the concurrent requests counter is zero.

Time to sleep now. I'll see if this still seems to be a good solution 
when waking up and implement it, unless someone else is faster or has a 
better idea ;-)

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 13:42, Vadim Gritsenko wrote:
> Bertrand Delacretaz wrote:
>
>>>   touch 12011200 sitemap.xmap
>>>
>>> reproduces the disposed ComponentLocator problem!
>>
>>
>> Definitely reproducible, see new comments in
>> http://issues.apache.org/bugzilla/show_bug.cgi?id=27249
>
>
> But, as far as I can tell (what's the acronym for this? ;)), this is 
> not related (should not be related) to Pier's issue - he has 
> CocoonServlet.dispose() in the stack trace.

Wrong stack trace! :-) Sorry... That was while I was shutting down 
Cocoon :-( :-( Let me fish a right one out!

	Pier


Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Bertrand Delacretaz wrote:

>>   touch 12011200 sitemap.xmap
>>
>> reproduces the disposed ComponentLocator problem!
>
>
> Definitely reproducible, see new comments in
> http://issues.apache.org/bugzilla/show_bug.cgi?id=27249


But, as far as I can tell (what's the acronym for this? ;)), this is not 
related (should not be related) to Pier's issue - he has 
CocoonServlet.dispose() in the stack trace.

Vadim


Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
>   touch 12011200 sitemap.xmap
>
> reproduces the disposed ComponentLocator problem!

Definitely reproducible, see new comments in
http://issues.apache.org/bugzilla/show_bug.cgi?id=27249

-Bertrand


Re: VERY odd exceptions...

Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 25 mai 04, à 13:22, Vadim Gritsenko a écrit :

> Sylvain Wallez wrote:
> ...
>
>> Bertrand's problem related to CVS $Id$ in comments makes me think to 
>> clock synchronisation problems between the CVS server and the Cocoon 
>> server (e.g. modification date moved back in time).
>
>
> I noticed sometime ago that setting sub-sitemap (not root sitemap!) 
> modification time somewhere in the future (or rewinding system clock 
> back) breaks system completely. I had no enough time to find a reason 
> though...

bingo:
   touch 12011200 sitemap.xmap

reproduces the disposed ComponentLocator problem!

-Bertrand


Re: VERY odd exceptions...

Posted by Joerg Heinicke <jo...@gmx.de>.
On 25.05.2004 13:22, Vadim Gritsenko wrote:

>> Bertrand's problem related to CVS $Id$ in comments makes me think to 
>> clock synchronisation problems between the CVS server and the Cocoon 
>> server (e.g. modification date moved back in time).
> 
> I noticed sometime ago that setting sub-sitemap (not root sitemap!) 
> modification time somewhere in the future (or rewinding system clock 
> back) breaks system completely. I had no enough time to find a reason 
> though...

*puuh* Thank you very much for the hint. The last days I worked with a 
Cocoon version from CVS from the beginning of the code freeze and tested 
with it extensively. I updated today to the released version, thought I 
could simply use it, deployed it on the live server - and got these odd 
exceptions. In the noon I read the first 5 mails, but did not care much 
about them as the problem did not occur for me - and now I was hit? So I 
read on and finally the touch on the sitemap was the solution for the 
problem, the live server is some minutes back in time.

BTW, I have *only* a root sitemap and it happened though. While a simple 
pipeline (the home page, just aggregation of different files, 
transformer, serializer) works, all the next links caused the exception. 
These links go to a match calling flowscript and from there back to 
sitemap. Maybe this gives a hint on the root of the problem, something 
like "2 pipelines must be involved", "pipeline hash" or so.

Joerg

Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Sylvain Wallez wrote:
...

> Bertrand's problem related to CVS $Id$ in comments makes me think to 
> clock synchronisation problems between the CVS server and the Cocoon 
> server (e.g. modification date moved back in time).


I noticed sometime ago that setting sub-sitemap (not root sitemap!) 
modification time somewhere in the future (or rewinding system clock 
back) breaks system completely. I had no enough time to find a reason 
though...

Vadim



Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Pier Fumagalli wrote:

> I'm having a bunch of problems with Avalon in production... And I  
> suppose they're kinda all interconnected.
>
> I get some odd ConcurrentModificationException(s) and  
> NullPointerException(s) all coming out from the component manager  
> "release/dispose" methods: I suppose that, given that the  
> ConcurrentModificationException comes out from my server log, 
> Excalibur  catches it, and it tries to dispose of a component, but at 
> this point  somehow somewhere it will throw a NullPointerException 
> (uncaught, as it  comes out in my System.err log)...
>
> Now, after this sequence of exceptions, from time to time, I start  
> getting things like IllegalStateException(s) mentioning that I cannot  
> lookup components on a disposed ComponentLocator...
>
> And this is BAD, because to get this working again, I have to SHUT 
> DOWN  COCOON, clean up my sitemaps and cocoon.xconf (I mean, removing 
> them,  and reverting back to my CVS version, don't ask my why, I don't 
> know),  restart it with these new "files" (which didn't change), and 
> the  exception goes away...


Bertrand's problem related to CVS $Id$ in comments makes me think to 
clock synchronisation problems between the CVS server and the Cocoon 
server (e.g. modification date moved back in time).

But what you describe here is really weird: there's absolutely no data 
related to sitemap or component manager that is persisted to disk.

Also, is your problem occuring when nothing changes in the system, or is 
it related to changes in the sitemap, like Bertrand's case?

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Stefano Mazzocchi <st...@apache.org>.
Pier Fumagalli wrote:

> Well, my only problem is that the whole "shabang" is pooled to death at 
> the moment.

have you tried setting all the pool sizes to zero? I know it's hacky, 
but might solve the issue and remove some of the pressure for now while 
we look for a more reasonable solution.

-- 
Stefano.


Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Sylvain Wallez wrote:

> Pier Fumagalli wrote:
>
> <snip/>
>
>> I patched Excalibur, and Bertrand is checking out if my patch solves 
>> also his bug... Now, if my patch works, given that this is not a bug 
>> in our code, how do we go in sticking that into the Excalibur CVS and 
>> re-release now that community of that project has collapsed?
>
>
> Fork it in Cocoon's repository?


-1. It's still in CVS, and Berin gave more reasons to avoid forking.

Vadim


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 13:18, Sylvain Wallez wrote:
> Pier Fumagalli wrote:
>
> <snip/>
>
>> I patched Excalibur, and Bertrand is checking out if my patch solves 
>> also his bug... Now, if my patch works, given that this is not a bug 
>> in our code, how do we go in sticking that into the Excalibur CVS and 
>> re-release now that community of that project has collapsed?
>
>
> Fork it in Cocoon's repository

-1000 :-(

	Pier


Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Pier Fumagalli wrote:

<snip/>

> I patched Excalibur, and Bertrand is checking out if my patch solves 
> also his bug... Now, if my patch works, given that this is not a bug 
> in our code, how do we go in sticking that into the Excalibur CVS and 
> re-release now that community of that project has collapsed?


Fork it in Cocoon's repository?

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Stefano Mazzocchi <st...@apache.org>.
Antonio Gallardo wrote:

> Stefano Mazzocchi dijo:
> 
>>There is a proposal for the creation of an Excalibur TLP that will be
>>discussed tomorrow by the ASF board. If that passes, I would expect
>>things to run a little more smoothly on the community side.
> 
> 
> Nothing against Excalibur and other projects try to be TLP. Seems to be a
> fever inside the ASF? Why?

For a long time, the ASF directors believed that "project containment" 
would increase communication and simplify incubation.

The Gump project showed that there was little (if any) correspondance 
between the dependencies and the containment, this (and the election of 
directors coming from the non-httpd world) yield as a result the 
suggestion that the ASF should suggest all project to be run like HTTPD, 
which is: no containment and the PMC is composed by the people who care 
about the code.

At the same time, the ASF board does *NOT* enforce this practice, it 
just suggests it.

> Antonio is just looking for an explanation, he is not trying to fight.

The ultimate reason why containment is worse than top level is that PMC 
are the organisms that provide technical leadership to the various 
project and, more important, provide legal and community oversight.

When projects were too contained (for example think of maven starting in 
turbine hosted in jakarta) the board had *no* clue of what was going on.

This is potentially harmful.

The ASF board is ultimately responsible for legal issues, but it cannot 
scale to oversee the entire thing, so that's why the board is normally 
very happy to create new top level projects *if* the necessary community 
requirements of solidity and diversity are met.

It's a basic form of delegation to the ones who really care and should 
really do the job of managing and overseeing the project work and the 
less layers between the top and the bottom, the better, the lighter and 
the more efficient.

This *does* leave the question on what happens when too many top level 
projects are created, and I think this is something that the currrent 
board will have to face.

Hope this helps.

-- 
Stefano.


Re: VERY odd exceptions...

Posted by Antonio Gallardo <ag...@agssa.net>.
Stefano Mazzocchi dijo:
> There is a proposal for the creation of an Excalibur TLP that will be
> discussed tomorrow by the ASF board. If that passes, I would expect
> things to run a little more smoothly on the community side.

Nothing against Excalibur and other projects try to be TLP. Seems to be a
fever inside the ASF? Why?

Antonio is just looking for an explanation, he is not trying to fight.

Best Regards,

Antonio Gallardo


Re: VERY odd exceptions...

Posted by Stefano Mazzocchi <st...@apache.org>.
Pier Fumagalli wrote:

> Now, if my patch works, given that this is not a bug in 
> our code, how do we go in sticking that into the Excalibur CVS and 
> re-release now that community of that project has collapsed?

There is a proposal for the creation of an Excalibur TLP that will be 
discussed tomorrow by the ASF board. If that passes, I would expect 
things to run a little more smoothly on the community side.

-- 
Stefano.


Re: VERY odd exceptions...

Posted by Berin Loritsch <bl...@d-haven.org>.
Pier Fumagalli wrote:

> 
> I patched Excalibur, and Bertrand is checking out if my patch solves 
> also his bug... Now, if my patch works, given that this is not a bug in 
> our code, how do we go in sticking that into the Excalibur CVS and 
> re-release now that community of that project has collapsed?
> 

Stay tuned.  The Excalibur stuff--including ECM--is proposed to be
revived in its own TLP.  There will be a place to put it there.
For now, the codebase has been migrated to SVN and is still within
the Avalon repository.


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 12:42, Gianugo Rabellino wrote:
> On May 25, 2004, at 1:30 PM, Pier Fumagalli wrote:
>> On 25 May 2004, at 11:22, Ugo Cei wrote:
>>> Pier Fumagalli wrote:
>>>>
>>>> I seriously think (I can't see it but can definitely smell it) that 
>>>> it  happens because one of those handlers (somehow) gets recycled, 
>>>> and it's  reused before it's actually fully disposed...
>>>> So, I _seriously_ think that by fixing the problem adding the  
>>>> appropriate synchronizations, we'll not solve a deeper bug 
>>>> somewhere in  the pools...
>>>
>>> I am more and more inclined to think that pooling is harmful more 
>>> than it is useful. Components shouldn't be pooled by default, only 
>>> on special occasions and after careful consideration. Apparently, 
>>> doing a bug-free pool is harder than it seems.
>>
>> Oh, well... I totally agree with you on this... Especially since the 
>> JVMs got quite a lot faster in the last few years... Pooling JDBC 
>> connections is one thing, pooling random objects another.
>
> But, again, you should consider that in quite a few occasions you're 
> pooling coonfigured & initialized objects. It's not just creation 
> overhead: at times configuration can be a complex beast to manage, 
> leading to heavy processing times.

If they are such beasts to configure, then they should also be made 
thread safe, and used as singletons...

For example, in JAXP, the "Template" is ThreadSafe, and can be used as 
a singleton by a bunch of "Transformer" instances all "configured" with 
the same stylesheet...

If you need to pool non-thread-safe object instances, then (IMVHO) 
you're doing something _REALLY_ wrong! :-P

ANYHOW...................

					- O - O - O - O - O -

I patched Excalibur, and Bertrand is checking out if my patch solves 
also his bug... Now, if my patch works, given that this is not a bug in 
our code, how do we go in sticking that into the Excalibur CVS and 
re-release now that community of that project has collapsed?

	Pier

Re: VERY odd exceptions...

Posted by Sylvain Wallez <sy...@apache.org>.
Gianugo Rabellino wrote:

>
> On May 25, 2004, at 1:30 PM, Pier Fumagalli wrote:
>
>> On 25 May 2004, at 11:22, Ugo Cei wrote:
>>
>>> Pier Fumagalli wrote:
>>>
>>>>
>>>> I seriously think (I can't see it but can definitely smell it) that 
>>>> it  happens because one of those handlers (somehow) gets recycled, 
>>>> and it's  reused before it's actually fully disposed...
>>>> So, I _seriously_ think that by fixing the problem adding the  
>>>> appropriate synchronizations, we'll not solve a deeper bug 
>>>> somewhere in  the pools...
>>>
>>>
>>> I am more and more inclined to think that pooling is harmful more 
>>> than it is useful. Components shouldn't be pooled by default, only 
>>> on special occasions and after careful consideration. Apparently, 
>>> doing a bug-free pool is harder than it seems.
>>
>>
>> Oh, well... I totally agree with you on this... Especially since the 
>> JVMs got quite a lot faster in the last few years... Pooling JDBC 
>> connections is one thing, pooling random objects another.
>
>
> But, again, you should consider that in quite a few occasions you're 
> pooling coonfigured & initialized objects. It's not just creation 
> overhead: at times configuration can be a complex beast to manage, 
> leading to heavy processing times.


That's a very bad side effect of Avalon, which pratically requires most 
non thread-safe objects to be poolable because of the object setup 
overhead. And this setup phase is exactly the same for each created 
instance, which is nothing but waste of CPU.

You can read "Alternatives to Poolable" [1] where I discussed this and 
proposed "component fields", which I admit to be rather a workaround 
than a real solution. The real solution is certainly to avoid 
non-treadsafe components as much as possible by using factory patterns 
like JAXP's Template/Transformer as outlined by Pier.

When designing components, it should be fairly easy to identify stateful 
services and then provide the actual service through instances created 
by a factory. Notice also that nothing prevents the factory to actually 
use an internal pool if needed. But only _if_ needed.

[1] http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=107640905422462&w=2

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }


Re: VERY odd exceptions...

Posted by Gianugo Rabellino <gi...@apache.org>.
On May 25, 2004, at 1:30 PM, Pier Fumagalli wrote:

> On 25 May 2004, at 11:22, Ugo Cei wrote:
>> Pier Fumagalli wrote:
>>>
>>> I seriously think (I can't see it but can definitely smell it) that 
>>> it  happens because one of those handlers (somehow) gets recycled, 
>>> and it's  reused before it's actually fully disposed...
>>> So, I _seriously_ think that by fixing the problem adding the  
>>> appropriate synchronizations, we'll not solve a deeper bug somewhere 
>>> in  the pools...
>>
>> I am more and more inclined to think that pooling is harmful more 
>> than it is useful. Components shouldn't be pooled by default, only on 
>> special occasions and after careful consideration. Apparently, doing 
>> a bug-free pool is harder than it seems.
>
> Oh, well... I totally agree with you on this... Especially since the 
> JVMs got quite a lot faster in the last few years... Pooling JDBC 
> connections is one thing, pooling random objects another.

But, again, you should consider that in quite a few occasions you're 
pooling coonfigured & initialized objects. It's not just creation 
overhead: at times configuration can be a complex beast to manage, 
leading to heavy processing times.

Then again, +1 on not using pools as plain "new" surrogates.

Ciao,
-- 
Gianugo Rabellino
Pro-netics s.r.l. -  http://www.pro-netics.com
Orixo, the XML business alliance: http://www.orixo.com


Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 May 2004, at 11:22, Ugo Cei wrote:
> Pier Fumagalli wrote:
>>
>> I seriously think (I can't see it but can definitely smell it) that 
>> it  happens because one of those handlers (somehow) gets recycled, 
>> and it's  reused before it's actually fully disposed...
>> So, I _seriously_ think that by fixing the problem adding the  
>> appropriate synchronizations, we'll not solve a deeper bug somewhere 
>> in  the pools...
>
> I am more and more inclined to think that pooling is harmful more than 
> it is useful. Components shouldn't be pooled by default, only on 
> special occasions and after careful consideration. Apparently, doing a 
> bug-free pool is harder than it seems.

Oh, well... I totally agree with you on this... Especially since the 
JVMs got quite a lot faster in the last few years... Pooling JDBC 
connections is one thing, pooling random objects another.

> I'm sorry if this isn't of any help to you in this situation, Pier.

Well, my only problem is that the whole "shabang" is pooled to death at 
the moment.

	Pier


Re: VERY odd exceptions...

Posted by Ugo Cei <u....@cbim.it>.
Pier Fumagalli wrote:
> I seriously think (I can't see it but can definitely smell it) that it  
> happens because one of those handlers (somehow) gets recycled, and it's  
> reused before it's actually fully disposed...
> 
> So, I _seriously_ think that by fixing the problem adding the  
> appropriate synchronizations, we'll not solve a deeper bug somewhere in  
> the pools...

I am more and more inclined to think that pooling is harmful more than 
it is useful. Components shouldn't be pooled by default, only on special 
occasions and after careful consideration. Apparently, doing a bug-free 
pool is harder than it seems.

I'm sorry if this isn't of any help to you in this situation, Pier.

	Ugo



Re: VERY odd exceptions...

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Pier Fumagalli wrote:

> (CRAP CRAP CRAP CRAP CRAP)


First, breathe!!! Now, think: Why in hell do you want to destroy Cocoon 
servlet on running production site??? I'm referring to line from your 
stacktrace:
    org.apache.cocoon.servlet.CocoonServlet.destroy(CocoonServlet.java:497)

And to answer your question about synchronization, actually, I think, 
synchronization in dispose() is not required, as to this moment, servlet 
should have just one thread - the thread which disposes the servlet.

Vadim



Re: VERY odd exceptions...

Posted by Pier Fumagalli <pi...@betaversion.org>.
Ok, let's shoot them one by one... The first one I can easily spot is  
the ConcurrentModificationException: relevant part of the stacktrace:

java.util.ConcurrentModificationException
         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:782)
         at java.util.HashMap$KeyIterator.next(HashMap.java:818)
         at  
org.apache.avalon.excalibur.component.ExcaliburComponentManager.dispose( 
ExcaliburComponentManager.java:607)
         at  
org.apache.cocoon.components.CocoonComponentManager.dispose(CocoonCompon 
entManager.java:468)
         at  
org.apache.avalon.framework.container.ContainerUtil.dispose(ContainerUti 
l.java:345)
         at org.apache.cocoon.Cocoon.dispose(Cocoon.java:489)

Now, let's see code snippet...

602 while( m_componentHandlers.size() > 0 )
603 {
604   for( Iterator iterator = m_componentHandlers.keySet().iterator();
605        iterator.hasNext(); )
606   {
607     final Object role = iterator.next();

Ok, this call is enclosed in a "synchronized(this)" statement, BUT:

At line 246 I have a m_componentHandlers.put(...) call which is not  
synchronized (might be a cause of the exception on the Iterator), at  
line 752 I have a m_componentHandlers.remove(...) on an iterator  
(????????????????????) which might cause MORE troubles, and definitely  
addComponent at line 847 where there is a m_componentHandlers.put(...)  
could be another culprit.

I seriously think (I can't see it but can definitely smell it) that it  
happens because one of those handlers (somehow) gets recycled, and it's  
reused before it's actually fully disposed...

So, I _seriously_ think that by fixing the problem adding the  
appropriate synchronizations, we'll not solve a deeper bug somewhere in  
the pools...

(CRAP CRAP CRAP CRAP CRAP)

	Pier