You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Derek Hohls <dh...@csir.co.za> on 2005/05/09 12:14:55 UTC
Cinclude transforms very *slow* [ was Re: Speeding up an
xinclude transform?]
Chris
Yes... the CInclude transformer output isn't cached, and I have
found that the approach you suggested is only marginally faster
than using the xinclude approach I had adopted originally.
I guess that working with information from numerous XML files
on disc is not really a viable operation using Cocoon... which is,
I think, a pity.
In my case its impossible to predict when content can be changed...
could be a whole lot of changes in a few minutes, and then
nothing for a few months. This makes it very hard to simply set
a time parameter as suggested in the documents. There needs to
be something a little more straightforward than that to make it
usable/useful.
Any more thoughts?
Derek
>>> cocoon@chrismaloney.com 2005/05/07 04:17:27 AM >>>
I misspoke in my last email (actually, mis-wrote :)
The CInclude transformer output isn't, by default, cached, but the
input
to it, "cocoon:/101.meta.xml", below, would be. In your case, where
you
have fifty or so inputs of that form, that's what you'd want. However,
you can also get the CInclude transformer to cache its output, as
described here:
http://cocoon.apache.org/2.1/userdocs/transformers/cinclude-transformer.html#Caching
I agree when you said "surely its Cocoon's "job" to check ....", but
I've gotten myself into a bit of a mess, with lots of these cached
pipeline fragments daisy-chained. I've also implemented "preemptive
caching" on some of them, and now I just haven't spent the time to dig
in and see where the cache isn't getting properly invalidated.
Cheers!
Derek Hohls wrote:
>Chris(?)
>
>Thanks - I will give this a try (cannot be slower than what I have
>now and looks pretty straightforward). It had not occurred to me
>because it seemed this would require 50+ calls to the same
>pipeline.. instead of just one pass through one... but I'll check.
>
>Re the caching - surely its Cocoon's "job" to check the files
>being used (if they are static, as in this case) and send through
>the latest version... but I will run some tests and see what occurs.
>
>Thanks
>Derek
>
>
>
>>>>cocoon@chrismaloney.com 2005/05/06 02:44:53 PM >>>
>>>>
>>>>
>Is the XPath expression the same in every case ("//inml:ind/meta")?
>If so, then it would be easy to switch to using CInclude, which is
>cached:
>
> <file name='101.xml'>
> <ci:include src='cocoon:/101.meta.xml'/>
> </file>
>
>And then define a new pipeline to produce 101.meta.xml:
>
> <map:match pattern='*.meta.xml'>
> <map:generator src='{1}.xml'/>
> <map:transform src='pull-out-ind-meta.xslt'/>
> <map:serialize>
> </map:match>
>
>I'm pretty new to Cocoon, actually, and I've been using this
technique
>a
>lot,
>for example, to generate my nav bar. I'm not altogether happy with
it,
>
>though,
>mainly because I can't figure out how to control the cache -- i.e. to
>make sure
>that it gets invalidated whenever {1}.xml changes. But, it's pretty
>fast.
>
>
>Derek Hohls wrote:
>
>
>
>>I am looking to find a way to speed up a key step in a pipeline:
>>
>>The one in question has the following steps:
>>
>><map:match pattern="ind-list">
>> <map:parameter name="handler" value="myindhandler"/>
>> <map:generate src="inds" type="directory" include="*.xml"/>
>> <map:transform src="stylesheets/ind/ind-xincludes.xsl" >
>> <map:parameter name="ind-dir" value="inds"/>
>> </map:transform>
>> <!-- *NOW SLOW* -->
>> <map:transform type="xinclude"/>
>> <map:serialize type="xml"/>
>></map:match>
>>
>>The pipeline is fast up to the end of the first transform,
>>resulting in XML which contains a number of tags of
>>the form:
>>
>><file name="101.xml">
>> <xi:include
>>href="inds/101.xml#xmlns(inml=http://www.myschema.com)xpointer(//inml:ind/meta)"/>
>></file>
>>
>>The number of tags varies by directory, but is typically about 50.
>>The files themselves are small - about 50k - and the "meta"
>>tags have only a few bytes of text.
>>
>>However, this last step takes over a minute! on a fast server
>>(2Gb memory, 3Ghz processor)...
>>
>>What can I do to ensure that this is speeded up significantly...?
>>ie at least a factor of 10!
>>
>>Thanks
>>Derek
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>
>>
>>
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>For additional commands, e-mail: users-help@cocoon.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>For additional commands, e-mail: users-help@cocoon.apache.org
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Cinclude transforms very *slow* [ was Re: Speeding up an xinclude
transform?]
Posted by Marc Salvetti <ma...@notremanou.net>.
Hello,
i had pretty much the same problem, a large number of small files to
aggregate, and i finally found the xpathdirectorygenerator, which is
pretty straightforward to use, doesn't have the caching issues of
document() function, and is faster than the technique of xinclude and
cinclude on a large number of files.
Hope this help,
Marc
Derek Hohls a écrit :
>Chris
>
>Yes... the CInclude transformer output isn't cached, and I have
>found that the approach you suggested is only marginally faster
>than using the xinclude approach I had adopted originally.
>
>I guess that working with information from numerous XML files
>on disc is not really a viable operation using Cocoon... which is,
>I think, a pity.
>
>In my case its impossible to predict when content can be changed...
>could be a whole lot of changes in a few minutes, and then
>nothing for a few months. This makes it very hard to simply set
>a time parameter as suggested in the documents. There needs to
>be something a little more straightforward than that to make it
>usable/useful.
>
>Any more thoughts?
>
>Derek
>
>
>
>>>>cocoon@chrismaloney.com 2005/05/07 04:17:27 AM >>>
>>>>
>>>>
>I misspoke in my last email (actually, mis-wrote :)
>The CInclude transformer output isn't, by default, cached, but the
>input
>to it, "cocoon:/101.meta.xml", below, would be. In your case, where
>you
>have fifty or so inputs of that form, that's what you'd want. However,
>
>you can also get the CInclude transformer to cache its output, as
>described here:
>http://cocoon.apache.org/2.1/userdocs/transformers/cinclude-transformer.html#Caching
>
>
>I agree when you said "surely its Cocoon's "job" to check ....", but
>I've gotten myself into a bit of a mess, with lots of these cached
>pipeline fragments daisy-chained. I've also implemented "preemptive
>caching" on some of them, and now I just haven't spent the time to dig
>
>in and see where the cache isn't getting properly invalidated.
>
>Cheers!
>
>Derek Hohls wrote:
>
>
>
>>Chris(?)
>>
>>Thanks - I will give this a try (cannot be slower than what I have
>>now and looks pretty straightforward). It had not occurred to me
>>because it seemed this would require 50+ calls to the same
>>pipeline.. instead of just one pass through one... but I'll check.
>>
>>Re the caching - surely its Cocoon's "job" to check the files
>>being used (if they are static, as in this case) and send through
>>the latest version... but I will run some tests and see what occurs.
>>
>>Thanks
>>Derek
>>
>>
>>
>>
>>
>>>>>cocoon@chrismaloney.com 2005/05/06 02:44:53 PM >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>Is the XPath expression the same in every case ("//inml:ind/meta")?
>>If so, then it would be easy to switch to using CInclude, which is
>>cached:
>>
>> <file name='101.xml'>
>> <ci:include src='cocoon:/101.meta.xml'/>
>> </file>
>>
>>And then define a new pipeline to produce 101.meta.xml:
>>
>> <map:match pattern='*.meta.xml'>
>> <map:generator src='{1}.xml'/>
>> <map:transform src='pull-out-ind-meta.xslt'/>
>> <map:serialize>
>> </map:match>
>>
>>I'm pretty new to Cocoon, actually, and I've been using this
>>
>>
>technique
>
>
>>a
>>lot,
>>for example, to generate my nav bar. I'm not altogether happy with
>>
>>
>it,
>
>
>>though,
>>mainly because I can't figure out how to control the cache -- i.e. to
>>
>>
>
>
>
>>make sure
>>that it gets invalidated whenever {1}.xml changes. But, it's pretty
>>fast.
>>
>>
>>Derek Hohls wrote:
>>
>>
>>
>>
>>
>>>I am looking to find a way to speed up a key step in a pipeline:
>>>
>>>The one in question has the following steps:
>>>
>>><map:match pattern="ind-list">
>>><map:parameter name="handler" value="myindhandler"/>
>>><map:generate src="inds" type="directory" include="*.xml"/>
>>><map:transform src="stylesheets/ind/ind-xincludes.xsl" >
>>> <map:parameter name="ind-dir" value="inds"/>
>>></map:transform>
>>><!-- *NOW SLOW* -->
>>><map:transform type="xinclude"/>
>>><map:serialize type="xml"/>
>>></map:match>
>>>
>>>The pipeline is fast up to the end of the first transform,
>>>resulting in XML which contains a number of tags of
>>>the form:
>>>
>>><file name="101.xml">
>>><xi:include
>>>href="inds/101.xml#xmlns(inml=http://www.myschema.com)xpointer(//inml:ind/meta)"/>
>>></file>
>>>
>>>The number of tags varies by directory, but is typically about 50.
>>>The files themselves are small - about 50k - and the "meta"
>>>tags have only a few bytes of text.
>>>
>>>However, this last step takes over a minute! on a fast server
>>>(2Gb memory, 3Ghz processor)...
>>>
>>>What can I do to ensure that this is speeded up significantly...?
>>>ie at least a factor of 10!
>>>
>>>Thanks
>>>Derek
>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>>For additional commands, e-mail: users-help@cocoon.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>
>>
>>
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>For additional commands, e-mail: users-help@cocoon.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>For additional commands, e-mail: users-help@cocoon.apache.org
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: Cinclude transforms very *slow* [ was Re: Speeding up an xinclude
transform?]
Posted by Geert Josten <Ge...@daidalos.nl>.
Hi there,
I saw someone make a remark on the xpointer/xpath expression, but never saw a response to it. Have
you tried changing the // in //inml:ind/meta to a precise path? Or is that impossible? And how ofter
does that element occur? Would it be sufficient to stop after the first occurrence of inml:ind or
meta? You might want to use something like:
/root-elem[1]/sub-elem1[1]/sub2[3]/sub3[2]/inml:ind[meta][1]/meta[1]
I can't oversee whether your problem has to do with caching or not. But I've been processing 1500+
files from filesystem using Cocoon, filtering, selecting, merging and converting them all together.
And though the first call is slow, subsequest requests are fast.
I know for sure that // is a performance killer, so I think it is worth investigating..
Cheers,
Geert
Derek Hohls wrote:
> Chris
>
> Yes... the CInclude transformer output isn't cached, and I have
> found that the approach you suggested is only marginally faster
> than using the xinclude approach I had adopted originally.
>
> I guess that working with information from numerous XML files
> on disc is not really a viable operation using Cocoon... which is,
> I think, a pity.
>
> In my case its impossible to predict when content can be changed...
> could be a whole lot of changes in a few minutes, and then
> nothing for a few months. This makes it very hard to simply set
> a time parameter as suggested in the documents. There needs to
> be something a little more straightforward than that to make it
> usable/useful.
>
> Any more thoughts?
>
> Derek
>
>
>>>>cocoon@chrismaloney.com 2005/05/07 04:17:27 AM >>>
>
> I misspoke in my last email (actually, mis-wrote :)
> The CInclude transformer output isn't, by default, cached, but the
> input
> to it, "cocoon:/101.meta.xml", below, would be. In your case, where
> you
> have fifty or so inputs of that form, that's what you'd want. However,
>
> you can also get the CInclude transformer to cache its output, as
> described here:
> http://cocoon.apache.org/2.1/userdocs/transformers/cinclude-transformer.html#Caching
>
>
> I agree when you said "surely its Cocoon's "job" to check ....", but
> I've gotten myself into a bit of a mess, with lots of these cached
> pipeline fragments daisy-chained. I've also implemented "preemptive
> caching" on some of them, and now I just haven't spent the time to dig
>
> in and see where the cache isn't getting properly invalidated.
>
> Cheers!
>
> Derek Hohls wrote:
>
>
>>Chris(?)
>>
>>Thanks - I will give this a try (cannot be slower than what I have
>>now and looks pretty straightforward). It had not occurred to me
>>because it seemed this would require 50+ calls to the same
>>pipeline.. instead of just one pass through one... but I'll check.
>>
>>Re the caching - surely its Cocoon's "job" to check the files
>>being used (if they are static, as in this case) and send through
>>the latest version... but I will run some tests and see what occurs.
>>
>>Thanks
>>Derek
>>
>>
>>
>>
>>>>>cocoon@chrismaloney.com 2005/05/06 02:44:53 PM >>>
>>>>>
>>>>>
>>
>>Is the XPath expression the same in every case ("//inml:ind/meta")?
>>If so, then it would be easy to switch to using CInclude, which is
>>cached:
>>
>> <file name='101.xml'>
>> <ci:include src='cocoon:/101.meta.xml'/>
>> </file>
>>
>>And then define a new pipeline to produce 101.meta.xml:
>>
>> <map:match pattern='*.meta.xml'>
>> <map:generator src='{1}.xml'/>
>> <map:transform src='pull-out-ind-meta.xslt'/>
>> <map:serialize>
>> </map:match>
>>
>>I'm pretty new to Cocoon, actually, and I've been using this
>
> technique
>
>>a
>>lot,
>>for example, to generate my nav bar. I'm not altogether happy with
>
> it,
>
>>though,
>>mainly because I can't figure out how to control the cache -- i.e. to
>
>
>>make sure
>>that it gets invalidated whenever {1}.xml changes. But, it's pretty
>>fast.
>>
>>
>>Derek Hohls wrote:
>>
>>
>>
>>
>>>I am looking to find a way to speed up a key step in a pipeline:
>>>
>>>The one in question has the following steps:
>>>
>>><map:match pattern="ind-list">
>>><map:parameter name="handler" value="myindhandler"/>
>>><map:generate src="inds" type="directory" include="*.xml"/>
>>><map:transform src="stylesheets/ind/ind-xincludes.xsl" >
>>> <map:parameter name="ind-dir" value="inds"/>
>>></map:transform>
>>><!-- *NOW SLOW* -->
>>><map:transform type="xinclude"/>
>>><map:serialize type="xml"/>
>>></map:match>
>>>
>>>The pipeline is fast up to the end of the first transform,
>>>resulting in XML which contains a number of tags of
>>>the form:
>>>
>>><file name="101.xml">
>>><xi:include
>>>href="inds/101.xml#xmlns(inml=http://www.myschema.com)xpointer(//inml:ind/meta)"/>
>>></file>
>>>
>>>The number of tags varies by directory, but is typically about 50.
>>>The files themselves are small - about 50k - and the "meta"
>>>tags have only a few bytes of text.
>>>
>>>However, this last step takes over a minute! on a fast server
>>>(2Gb memory, 3Ghz processor)...
>>>
>>>What can I do to ensure that this is speeded up significantly...?
>>>ie at least a factor of 10!
>>>
>>>Thanks
>>>Derek
>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>>For additional commands, e-mail: users-help@cocoon.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
>>For additional commands, e-mail: users-help@cocoon.apache.org
>>
>>
>>
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
>
--
=====================================
NB: het Daidalos kantoor is sinds 22 april
jl. gevestigd op een nieuw adres:
Daidalos BV
Hoekeindsehof 1 - 4
2665 JZ Bleiswijk
tel: +31 (0)10 850 12 00
fax: +31 (0)10 850 11 99
Bovenstaand adres is tevens het postadres.
======================
Geert.Josten@Daidalos.nl
IT-consultant at Daidalos BV
http://www.daidalos.nl/
GPG: 1024D/12DEBB50
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org