You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by John Williams <jo...@ennui.co.za> on 2003/08/09 20:32:50 UTC
lastModificationDate misuse? - old thread - progress
I have been mystified by the symptoms of the problem raised by Stuart Roebuck some time ago on cocoon-dev and I can't see if/when it has been fixed or a workaround/hack avail:
If XSP is the result of a pipeline then the ServerPages Generator can't work out its currency and randomly re-generates and compiles. To make matters worse if the XSP has the code necessary for resultant objects to be cached then the code generated and compiled is not even used, ie a cached object is used instead.
I have inspected the code of 2.0.4 and it is still there, ie code is same as when reported in 2001. Surely a typical use of cocoon is to build XSP using XSLT?
John
Stuart Roebuck wrote:
>>On Tuesday, August 14, 2001, at 04:04 pm, Carsten Ziegeler wrote:
>>
>>> In SitemapSource.java are the lines:
>>>
>>>> // the event pipeline is cacheable
>>>> // now calculate a last modification date
>>>> String hashKey = pck.toString() + validity.toString();
>>>> this.lastModificationDate = HashUtil.hash(hashKey);
>>>
>>> this is used in ProgramGeneratorImpl.java:
>>>
>>>> /*
>>>> * FIXME: It's the program (not the instance) that must
>>>> * be queried for changes!!!
>>>> */
>>>> if (programInstance != null &&
>>>> programInstance.modifiedSince(source.getLastModified())) {
>>>> // Release the component.
>>>> release(programInstance);
>>>
>>> consequently, any use of the ServerPages Generator which takes an input
>>> from within the sitemap (e.g. using a "cocoon:/" URI) will result
>>> in (on a
>>> practically random basis) certain XSP (et. al) pages being regenerated
>>> over and over again, and will potentially prevent updated XSP (et. al)
>>> from being properly regenerated.
>>>
>> Hm, that is true. The problem is that I don't saw any other way to
>> implement the lastModificationDate for the SitemapSource than hashing
>> the key.
>> Usually this modification date is used in the caching algorithm
>> and is there only tested against equal. So there this hack works.
>>
>> Any solution for this?
>
>For the caching, I presume there is a mapping of hashKeys to
>serializedResults. If this is true, then couldn't these also map to a
>date-stamp of when the result was cached and have lastModificationDate =
>lastCachedDate?
Re: lastModificationDate misuse? - old thread - progress
Posted by John Williams <jo...@ennui.co.za>.
Carsten Ziegeler wrote:
> The "S-xml-1_=NOP Validity" is appended by the xml serializer, so
> in fact the cached entry with the "S-xml-1_=NOP Validity" is for
> a complete pipeline with serializer. I'm still wondering
> why this is the value for prev. For which internal pipeline
> does this happen?
Any/every pipeline that builds up the xsp.
I created a very simple pipeline (see below) to test and I got the same
problem, ie "S-xml-1_=NOP Validity" appearing in the prev. I cannot find
the code appends the "S-xml-1_=NOP Validity" as follows:
prev: {G-file--7461341140568259924_=TimeStamp Validity[1060277818000],
T-xslt-2097470703455153761_=TimeStamp Validity[1062494248000], S-xml-1_=NOP
Validity}
curr: {G-file--7461341140568259924_=TimeStamp Validity[1060277818000],
T-xslt-2097470703455153761_=TimeStamp Validity[1062494248000]}
>From sitemap
<map:match pattern="test-*.xml">
<map:generate type="serverpages" src="cocoon://crm/test-{1}.xsp"/>
<map:serialize type="xml"/>
</map:match>
<map:match pattern="test-*.xsp">
<map:generate src="tableDefs/{1}.xml"/>
<map:transform src="stylesheets/test.xsl"/>
<map:serialize type="xml"/>
</map:match>
my serialiser is: org.apache.cocoon.serialization.XMLSerializer
my serverpages generator is:
org.apache.cocoon.generation.ServerPagesGenerator
>From .xconf
my event-pipeline
class="org.apache.cocoon.components.pipeline.CachingEventPipeline
RE: lastModificationDate misuse? - old thread - progress
Posted by Carsten Ziegeler <cz...@s-und-n.de>.
John Williams wrote:
>
> Carsten Ziegeler wrote:
> > > Inspecting the contents of the currValidity and prevValidity
> Maps I find
> > > that they are equivalent excepting that the prevValidity contains an
> entry
> > > "S-xml-1_=NOP Validity" so the test fails and code is
> > > re-generated.
> >
> > Huh! That's strange, now as far as I remember the code, the prevValidity
> > should
> > not contain "S-xml-1_=NOP Validity", as this is the validity for the
> > serializer
> > which is not used for internal pipeline calls. Hmm.
>
> I assume you mean src="cocoon://foo" calls and not internal-only
> pipelines,
> ie one with internal-only="true".
Yes.
>
> I have a fairly deep pipeline underlying the generated XSP "file"
> from which
> .java is generated. Maybe this depth exposes a problem in discarding
> NOPCacheValidity for internal pipeline calls. Can you see from
> the following
> which class is adding the "S-xml-1_=NOP Validity", I will then
> have a poke
> around/debug.
>
>
> Below are some details of the case causing the problems:
>
> create-categories.xsp - which has the "S-xml-1_=NOP Validity" in the
> prevValidity is:
> {
> aggregation of {
>
> aggregation of {tableDefs/categories.xml,
> appDef/categories/UISpecifics.xml and appDef/terminologyMappings.xml}
> transformed with stylesheets/buildUIStructure.xsl
> and
> appDef/categories/aCustomisation.xml
> }
> transformed with stylesheets/applyCustomisation.xsl
> }
> transformed with stylesheets/create.xsl
>
> This is what validities look like: (equal besides the "S-xml-1_=NOP
> Validity")
>
> prev: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973],
> T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp
> Validity[1061982760000]:TimeStamp Validity[1061976654000]], S-xml-1_=NOP
> Validity}
>
> curr: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973],
> T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp
> Validity[1061982760000]:TimeStamp Validity[1061976654000]]}
>
> Relevant part of the sitemap is as follows:
>
> <map:match pattern="table-*_*.xml">
> <map:aggregate element="root">
> <map:part src="cocoon://crm/table-{1}.xml"/>
> <map:part src="appDef/{1}/{2}.xml" strip-root="true"/>
> </map:aggregate>
> <map:transform src="stylesheets/applyCustomisation.xsl"/>
> <map:serialize type="xml"/>
> </map:match>
>
> <map:match pattern="table-*.xml">
> <map:aggregate element="UIDef">
> <map:part src="tableDefs/{1}.xml"/>
> <map:part src="appDef/{1}/UISpecifics.xml"/>
> <map:part src="appDef/terminologyMappings.xml"/>
> </map:aggregate>
> <map:transform src="stylesheets/buildUIStructure.xsl"/>
> <map:serialize type="xml"/>
> </map:match>
>
> <map:match pattern="*-*.xsp">
> <map:generate src="cocoon://crm/table-{2}.xml"/>
> <map:transform src="stylesheets/{1}.xsl"/>
> <map:serialize type="xml"/>
> </map:match>
>
The "S-xml-1_=NOP Validity" is appended by the xml serializer, so
in fact the cached entry with the "S-xml-1_=NOP Validity" is for
a complete pipeline with serializer. I'm still wondering
why this is the value for prev. For which internal pipeline
does this happen?
Carsten
Re: lastModificationDate misuse? - old thread - progress
Posted by John Williams <jo...@ennui.co.za>.
Carsten Ziegeler wrote:
> > Inspecting the contents of the currValidity and prevValidity Maps I find
> > that they are equivalent excepting that the prevValidity contains an
entry
> > "S-xml-1_=NOP Validity" so the test fails and code is
> > re-generated.
>
> Huh! That's strange, now as far as I remember the code, the prevValidity
> should
> not contain "S-xml-1_=NOP Validity", as this is the validity for the
> serializer
> which is not used for internal pipeline calls. Hmm.
I assume you mean src="cocoon://foo" calls and not internal-only pipelines,
ie one with internal-only="true".
I have a fairly deep pipeline underlying the generated XSP "file" from which
.java is generated. Maybe this depth exposes a problem in discarding
NOPCacheValidity for internal pipeline calls. Can you see from the following
which class is adding the "S-xml-1_=NOP Validity", I will then have a poke
around/debug.
Below are some details of the case causing the problems:
create-categories.xsp - which has the "S-xml-1_=NOP Validity" in the
prevValidity is:
{
aggregation of {
aggregation of {tableDefs/categories.xml,
appDef/categories/UISpecifics.xml and appDef/terminologyMappings.xml}
transformed with stylesheets/buildUIStructure.xsl
and
appDef/categories/aCustomisation.xml
}
transformed with stylesheets/applyCustomisation.xsl
}
transformed with stylesheets/create.xsl
This is what validities look like: (equal besides the "S-xml-1_=NOP
Validity")
prev: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973],
T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp
Validity[1061982760000]:TimeStamp Validity[1061976654000]], S-xml-1_=NOP
Validity}
curr: {G-file-7747944243312186483_=TimeStamp Validity[1062058603973],
T-xslt--3690387136403106359_=Aggregated Validity[TimeStamp
Validity[1061982760000]:TimeStamp Validity[1061976654000]]}
Relevant part of the sitemap is as follows:
<map:match pattern="table-*_*.xml">
<map:aggregate element="root">
<map:part src="cocoon://crm/table-{1}.xml"/>
<map:part src="appDef/{1}/{2}.xml" strip-root="true"/>
</map:aggregate>
<map:transform src="stylesheets/applyCustomisation.xsl"/>
<map:serialize type="xml"/>
</map:match>
<map:match pattern="table-*.xml">
<map:aggregate element="UIDef">
<map:part src="tableDefs/{1}.xml"/>
<map:part src="appDef/{1}/UISpecifics.xml"/>
<map:part src="appDef/terminologyMappings.xml"/>
</map:aggregate>
<map:transform src="stylesheets/buildUIStructure.xsl"/>
<map:serialize type="xml"/>
</map:match>
<map:match pattern="*-*.xsp">
<map:generate src="cocoon://crm/table-{2}.xml"/>
<map:transform src="stylesheets/{1}.xsl"/>
<map:serialize type="xml"/>
</map:match>
RE: lastModificationDate misuse? - old thread - progress
Posted by Carsten Ziegeler <cz...@s-und-n.de>.
John Williams wrote:
>
> Joerg Heinicke wrote
> > Bug was fixed some days ago:
> > http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten
> > requested for testers of the patch, so if you can test the behaviour
> > with the current Cocoon 2.0 CVS it will be good.
> > > If XSP is the result of a pipeline then the ServerPages
> Generator can't
> > > work out its currency and randomly re-generates and compiles. To make
> > > I have inspected the code of 2.0.4 and it is still there, ie code is
> > > same as when reported in 2001. Surely a typical use of cocoon is to
> > > build XSP using XSLT?
>
> I have tested (with cocoon-2.0_20030812221701) and believe the problem has
> only been partially solved. Instead of the prior situation where
> the .java &
> .class were re-generated from XSP on a random basis - they now are always
> regenerated. This is at least consistent - but very wasteful on
> resources &
> latency in requests.
>
> I have investigated the source and found that the test as to
> whether or not
> regeneration takes place is:
> !currValidity.equals(prevValidity)
>
> Inspecting the contents of the currValidity and prevValidity Maps I find
> that they are equivalent excepting that the prevValidity contains an entry
> "S-xml-1_=NOP Validity" so the test fails and code is
> re-generated.
Huh! That's strange, now as far as I remember the code, the prevValidity
should
not contain "S-xml-1_=NOP Validity", as this is the validity for the
serializer
which is not used for internal pipeline calls. Hmm.
So, I think we should investigate why this validity object is there instead
of ignoring it later on.
Carsten
Re: lastModificationDate misuse? - old thread - progress
Posted by John Williams <jo...@ennui.co.za>.
Joerg Heinicke wrote
> Bug was fixed some days ago:
> http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten
> requested for testers of the patch, so if you can test the behaviour
> with the current Cocoon 2.0 CVS it will be good.
> > If XSP is the result of a pipeline then the ServerPages Generator can't
> > work out its currency and randomly re-generates and compiles. To make
> > I have inspected the code of 2.0.4 and it is still there, ie code is
> > same as when reported in 2001. Surely a typical use of cocoon is to
> > build XSP using XSLT?
I have tested (with cocoon-2.0_20030812221701) and believe the problem has
only been partially solved. Instead of the prior situation where the .java &
.class were re-generated from XSP on a random basis - they now are always
regenerated. This is at least consistent - but very wasteful on resources &
latency in requests.
I have investigated the source and found that the test as to whether or not
regeneration takes place is:
!currValidity.equals(prevValidity)
Inspecting the contents of the currValidity and prevValidity Maps I find
that they are equivalent excepting that the prevValidity contains an entry
"S-xml-1_=NOP Validity" so the test fails and code is re-generated. However,
it appears that the "NOP Validity" is simply indicating an always valid
condition - so code should not be re-generated.
I have added some code to strip the NOPCacheValidity entries and do a
comparison and my problem is solved - ie code is only regenerated when the
cached objects are out-of-date.
I have pasted the diff of my patch at the end of this message. I think the
bug http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348 should be
re-opened. Your views Joerg , Carsten?
John Williams
Suggested patch to SiteMapSource.java:
@@ -56,12 +56,15 @@
import java.io.InputStream;
import java.net.MalformedURLException;
import java.util.Map;
+import java.util.HashMap;
+import java.util.Iterator;
import org.apache.avalon.framework.component.ComponentException;
import org.apache.avalon.framework.component.ComponentManager;
import org.apache.cocoon.ProcessingException;
import org.apache.cocoon.Processor;
import org.apache.cocoon.caching.PipelineCacheKey;
+import org.apache.cocoon.caching.NOPCacheValidity;
import org.apache.cocoon.components.CocoonComponentManager;
import org.apache.cocoon.components.EnvironmentStack;
import org.apache.cocoon.components.pipeline.CacheableEventPipeline;
@@ -322,7 +325,7 @@
Map currValidity = cep.generateValidity(this.environment);
- if (prevValidity == null || !currValidity.equals(prevValidity)) {
+ if (prevValidity == null || !equalsIgnoreNOPCacheValidities(currValidity,
prevValidity)) {
// validity changed, update cached validity, timestamp and last modified
this.lastModificationDate = System.currentTimeMillis();
obj = new Object[] {new Long (this.lastModificationDate), currValidity};
@@ -365,6 +368,29 @@
this.needsRefresh = false;
}
+ static boolean equalsIgnoreNOPCacheValidities (Map aMapWithNOPValidities,
Map anotherMapWithNOPValidities) {
+ return
stripNOPValidities(aMapWithNOPValidities).toString().equals(stripNOPValiditi
es(anotherMapWithNOPValidities).toString());
+ }
+
+
+ static Map stripNOPValidities(Map withNOPValidities) {
+ if (withNOPValidities == null) return null;
+ else {
+ if (withNOPValidities.containsValue(NOPCacheValidity.CACHE_VALIDITY)) {
+ Map retVal = new HashMap();
+ Iterator iter = withNOPValidities.entrySet().iterator();
+ for (; iter.hasNext(); ) {
+ Map.Entry entry = (Map.Entry)iter.next();
+ if (!entry.getValue().equals(NOPCacheValidity.CACHE_VALIDITY))
+ retVal.put(entry.getKey(), entry.getValue());
+ }
+ return retVal;
+ }
+ else return withNOPValidities;
+ }
+ }
+
+
/**
* Return a new <code>InputSource</code> object
*/
Re: lastModificationDate misuse? - old thread - progress
Posted by Joerg Heinicke <jo...@gmx.de>.
Bug was fixed some days ago:
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14348. Carsten
requested for testers of the patch, so if you can test the behaviour
with the current Cocoon 2.0 CVS it will be good.
Joerg
John Williams wrote:
> I have been mystified by the symptoms of the problem raised by Stuart
> Roebuck some time ago on cocoon-dev and I can't see if/when it has been
> fixed or a workaround/hack avail:
> If XSP is the result of a pipeline then the ServerPages Generator can't
> work out its currency and randomly re-generates and compiles. To make
> matters worse if the XSP has the code necessary for resultant objects to
> be cached then the code generated and compiled is not even used, ie a
> cached object is used instead.
>
> I have inspected the code of 2.0.4 and it is still there, ie code is
> same as when reported in 2001. Surely a typical use of cocoon is to
> build XSP using XSLT?
>
> John
>
>
> Stuart Roebuck wrote:
> >>On Tuesday, August 14, 2001, at 04:04 pm, Carsten Ziegeler wrote:
> >>
> >>> In SitemapSource.java are the lines:
> >>>
> >>>> // the event pipeline is cacheable
> >>>> // now calculate a last modification date
> >>>> String hashKey = pck.toString() + validity.toString();
> >>>> this.lastModificationDate = HashUtil.hash(hashKey);
> >>>
> >>> this is used in ProgramGeneratorImpl.java:
> >>>
> >>>> /*
> >>>> * FIXME: It's the program (not the instance) that must
> >>>> * be queried for changes!!!
> >>>> */
> >>>> if (programInstance != null &&
> >>>> programInstance.modifiedSince(source.getLastModified())) {
> >>>> // Release the component.
> >>>> release(programInstance);
> >>>
> >>> consequently, any use of the ServerPages Generator which takes an input
> >>> from within the sitemap (e.g. using a "cocoon:/" URI) will result
> >>> in (on a
> >>> practically random basis) certain XSP (et. al) pages being regenerated
> >>> over and over again, and will potentially prevent updated XSP (et. al)
> >>> from being properly regenerated.
> >>>
> >> Hm, that is true. The problem is that I don't saw any other way to
> >> implement the lastModificationDate for the SitemapSource than hashing
> >> the key.
> >> Usually this modification date is used in the caching algorithm
> >> and is there only tested against equal. So there this hack works.
> >>
> >> Any solution for this?
> >
> >For the caching, I presume there is a mapping of hashKeys to
> >serializedResults. If this is true, then couldn't these also map to a
> >date-stamp of when the result was cached and have lastModificationDate =
> >lastCachedDate?