You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Grzegorz Kossakowski <gr...@tuffmail.com> on 2007/11/17 14:38:44 UTC
Deprecation of CInclude transformer
Hello
Basing on Vadim's explanation[1]:
"cinclude is old code, ready to be deprecated. include is new code, with new
features (like parallel includes), and xinclude is based on a standard. If you
can add xinclude into include transformer, feel free to deprecate xinclude."
I would like to hear your opinions on deprecating CInclude transformer. The arguments for
deprecating this transformer are pretty obvious:
1. There are three different transformers for the same purpose, we should remove some of them
2. There is a transformer (Include) that is meant to replace old, CInclude transformer. So there is
a replacement for the stuff I would like to deprecate.
Anyone would object if I was going to start a vote?
[1] http://article.gmane.org/gmane.text.xml.cocoon.devel/75885
--
Grzegorz Kossakowski
Committer and PMC Member of Apache Cocoon
http://reflectingonthevicissitudes.wordpress.com/
RE: Deprecation of CInclude transformer
Posted by Ard Schrijvers <a....@hippo.nl>.
> Ralph Goers wrote:
> Actually, this may be exactly the reason my developer
> couldn't get the IncludeTransformer to work. I'll have to
> have hime take a look at it again.
If he still has problems let me know, I might be able to help you out.
Regards Ard
>
> Ralph
>
Re: Deprecation of CInclude transformer
Posted by Ralph Goers <Ra...@dslextreme.com>.
Ard Schrijvers wrote:
>> Vadim Gritsenko wrote:
>> Indeed. I think this is a failure on CocoonSource part.
>> CocoonSource content depends on the sitemap.xmap, but
>> CocoonSource validity does not include validity of the
>> sitemap. As a result, even if sitemap is changed,
>> CocoonSource validity stays valid.
>>
>> I'm not sure though if I want to have this bug fixed - it
>> would add more overhead...
>>
>
> Don't think either it is worth the trouble. You can work years with it
> without ever noticing, it is only happening during development when you
> happen to change something in a way described above (which will hardly
> ever happen)
>
Actually, this may be exactly the reason my developer couldn't get the
IncludeTransformer to work. I'll have to have hime take a look at it again.
Ralph
RE: Deprecation of CInclude transformer
Posted by Ard Schrijvers <a....@hippo.nl>.
> Vadim Gritsenko wrote:
>
> Have you seen this part?
>
> return key == null? "I": "I" + key;
> }
>
I never realized this specifically, but used to solve it similar (see
below)
>
> All you need to do is to pass in extra key:
>
> <map:match pattern="main">
> <map:generate src="someXML.xml"/>
> <map:transform type="include">
> <map:parameter name="key" value="{request-param:strange}"/>
> </map:transform>
> <map:serialize/>
> </map:match>
True, and as a matter of fact, you can put the key in *any* of the
generators (if they put the parameters in the cachekey) *or* any of the
transformers (which i used to do, but in the include transformer makes
more sense) (again if the honour the parameters in the cachekey
ofcourse). All that matters is that the parameter gets accounted for.
But still, there are other cases where you might not get away with this
'adding a key parameter'. Think about the usecase that instead of some
selector with a query-param, you use a resource-exist-selector. You
cannot account for this in the include transformer with a parameter (or
i think you have to add this input module...though it may be around
already. I can't remember from the top of my head)
>
> Hopefully information above helps you to keep them caching :)
>
I would certainly recommend so :-) The thing I wanted to point out, is
that when using the IncludeTransformer, you might need to really
understand how cocoon's caching work, and how cachekeys are being
created. It is in fact really simple, but I have just seen to many times
pipelines like these:
<map:pipeline type="caching">
<map:match pattern="common-used-part">
<map:generate src="foo.xml"/>
<map:transform src="bar.xsl>
<map:parameter name="{date:yyyyMMddHHddmmss}"/>
</map:transform/>
</map:match/>
</map:pipeline>
total useless cache....or my favorite:
<map:pipeline type="caching">
<map:match pattern="common-used-part">
<map:generate src="foo.xml"/>
<map:transform src="bar.xsl>
<map:parameter name="{header:Referer}"/>
</map:transform/>
</map:match/>
</map:pipeline>
Anyway, bottomline is that the InludeTransformer is IMO definitely the
one to use.
Regards Ard
Re: Deprecation of CInclude transformer
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Ard Schrijvers wrote:
>> Vadim Gritsenko wrote:
>> Indeed. I think this is a failure on CocoonSource part.
>> CocoonSource content depends on the sitemap.xmap, but
>> CocoonSource validity does not include validity of the
>> sitemap. As a result, even if sitemap is changed,
>> CocoonSource validity stays valid.
>>
>> I'm not sure though if I want to have this bug fixed - it
>> would add more overhead...
>
> Don't think either it is worth the trouble. You can work years with it
> without ever noticing, it is only happening during development when you
> happen to change something in a way described above (which will hardly
> ever happen)
Nod. Agree.
>>> Also, if the second pipeline /fetchSnippet start with a map:select
>>> first, the things are getting even complexer (the little catch)....if
>>> somebody wants to know about it I can elaborate.
>> Fire away! ;-)
>
> Okay, you asked for it :-)
>
> Suppose I have an include :
>
> <include:include src="cocoon://fetchSnippet"/>
>
> and I have a pipeline :
>
> <map:selectors default="parameter">
> <map:selector name="simple"
> src="org.apache.cocoon.selection.SimpleSelector"/>
> </map:selectors>
>
> <map:match pattern="fetchSnippet">
> <map:select type="simple">
> <map:parameter name="value"
> value="{request-param:strange}"/>
> <map:when test="true">
> <map:generate src="test1.xml"/>
> </map:when>
> <map:otherwise>
> <map:generate src="test2.xml"/>
> </map:otherwise>
> </map:select>
> <map:serialize type="xml"/>
> </map:match>
>
> Now, when your first call is for example : /main?strange=true
>
> and the main pipeline is again
>
> <map:match pattern="main">
> <map:generate src="someXML.xml"/>
> <map:transform type="include"/>
> <map:serialize/>
> </map:match>
>
> You will have included test1.xml, since {request-param:strange}
> evaluates to 'true'. Now, calling
> /main?strange=false will *not* give you test2.xml as the include,
> because, the main pipeline was perfectly cacheable, the 'stange'
> parameter was not included in its cachekey and the validity object is
> valid since no src whatsoever changed. Touching again test1.xml and
> calling again returns you test2.xml. Calling /main?strange=true
> afterwards gives you again test2.xml.
:)
Have you seen this part?
/**
* <p>Return the caching key associated with this transformation.</p>
*
* <p>When including <code>cocoon://</code> sources with dynamic
* content depending on environment (request parameters, session attributes,
* etc), it makes sense to provide such environment values to the transformer
* to be included into the key using <code>key</code> sitemap parameter.</p>
*
* @see CacheableProcessingComponent#getKey()
*/
public Serializable getKey() {
/*
* In case of including "cocoon://" or other dynamic sources key
* ideally has to include ProcessingPipelineKey of the included
* "cocoon://" sources, but it's not possible as at this time
* we don't know yet which sources will get included into the
* response.
*
* Hence, javadoc recommends providing key using sitemap parameter.
*/
return key == null? "I": "I" + key;
}
> You might consider it undesirable/incorrect behavior, but I just learned
> to use see the include transformer as if you add some pipeline parts
> from another pipe in your pipeline. So, if you have an include involving
> a map:select, you might see as if you were adding the pipe parts which
> are resolved by the map:select.
All you need to do is to pass in extra key:
<map:match pattern="main">
<map:generate src="someXML.xml"/>
<map:transform type="include">
<map:parameter name="key" value="{request-param:strange}"/>
</map:transform>
<map:serialize/>
</map:match>
> Also, the times I had to deal with this are few, but, OTOH, when I got
> the pleasure of debugging externally developed websites with a main
> sitemap of > 4.000 lines, and this was actually one of the problems, you
> can imagine how hard it is to find! :-)
>
>> BTW IncludeTransformer operation can not be changed easily.
>> If you try to resolve each included URI, to check its new key
>> and validity, you'd also have to parse (or store somewhere!)
>> included parts in order to resolve nested includes - if
>> recursive inclusion is switched on.
>
> Yes indeed. We certainly do not want that! It is working really fine
> ATM, but if you encounter one of the exceptional examples I
> described....well, then you either have to learn about cocoon caching,
> or put you pipelines to noncaching :-)
Hopefully information above helps you to keep them caching :)
>> IMHO POSTing stuff to a source does not really belong to a
>> include transformer, but rather to some other type of transformer...
>
> A PostTransformer :-)
Personally I'd not limit it to specific HTTP method, but allow to specify any
HTTP method, headers, and request body you want. HTTPRequestTransformer?
Vadim
Re: Deprecation of CInclude transformer
Posted by Joerg Heinicke <jo...@gmx.de>.
On 21.11.2007 8:47 Uhr, Ard Schrijvers wrote:
>> BTW IncludeTransformer operation can not be changed easily.
>> If you try to resolve each included URI, to check its new key
>> and validity, you'd also have to parse (or store somewhere!)
>> included parts in order to resolve nested includes - if
>> recursive inclusion is switched on.
>
> Yes indeed. We certainly do not want that! It is working really fine
> ATM, but if you encounter one of the exceptional examples I
> described....well, then you either have to learn about cocoon caching,
> or put you pipelines to noncaching :-)
But why not provide one feature-complete include transformer (but
without the POST and other non-include stuff) instead of two with
different feature sets? We could just add a configuration option that
checks validities recursively. Then we could really deprecate and remove
CInclude transformer.
Joerg
RE: Deprecation of CInclude transformer
Posted by Ard Schrijvers <a....@hippo.nl>.
> Vadim Gritsenko wrote:
> Indeed. I think this is a failure on CocoonSource part.
> CocoonSource content depends on the sitemap.xmap, but
> CocoonSource validity does not include validity of the
> sitemap. As a result, even if sitemap is changed,
> CocoonSource validity stays valid.
>
> I'm not sure though if I want to have this bug fixed - it
> would add more overhead...
Don't think either it is worth the trouble. You can work years with it
without ever noticing, it is only happening during development when you
happen to change something in a way described above (which will hardly
ever happen)
>
>
> > Also, if the second pipeline /fetchSnippet start with a map:select
> > first, the things are getting even complexer (the little
> catch)....if
> > somebody wants to know about it I can elaborate.
>
> Fire away! ;-)
Okay, you asked for it :-)
Suppose I have an include :
<include:include src="cocoon://fetchSnippet"/>
and I have a pipeline :
<map:selectors default="parameter">
<map:selector name="simple"
src="org.apache.cocoon.selection.SimpleSelector"/>
</map:selectors>
<map:match pattern="fetchSnippet">
<map:select type="simple">
<map:parameter name="value"
value="{request-param:strange}"/>
<map:when test="true">
<map:generate src="test1.xml"/>
</map:when>
<map:otherwise>
<map:generate src="test2.xml"/>
</map:otherwise>
</map:select>
<map:serialize type="xml"/>
</map:match>
Now, when your first call is for example : /main?strange=true
and the main pipeline is again
<map:match pattern="main">
<map:generate src="someXML.xml"/>
<map:transform type="include"/>
<map:serialize/>
</map:match>
You will have included test1.xml, since {request-param:strange}
evaluates to 'true'. Now, calling
/main?strange=false will *not* give you test2.xml as the include,
because, the main pipeline was perfectly cacheable, the 'stange'
parameter was not included in its cachekey and the validity object is
valid since no src whatsoever changed. Touching again test1.xml and
calling again returns you test2.xml. Calling /main?strange=true
afterwards gives you again test2.xml.
You might consider it undesirable/incorrect behavior, but I just learned
to use see the include transformer as if you add some pipeline parts
from another pipe in your pipeline. So, if you have an include involving
a map:select, you might see as if you were adding the pipe parts which
are resolved by the map:select.
Also, the times I had to deal with this are few, but, OTOH, when I got
the pleasure of debugging externally developed websites with a main
sitemap of > 4.000 lines, and this was actually one of the problems, you
can imagine how hard it is to find! :-)
>
> BTW IncludeTransformer operation can not be changed easily.
> If you try to resolve each included URI, to check its new key
> and validity, you'd also have to parse (or store somewhere!)
> included parts in order to resolve nested includes - if
> recursive inclusion is switched on.
Yes indeed. We certainly do not want that! It is working really fine
ATM, but if you encounter one of the exceptional examples I
described....well, then you either have to learn about cocoon caching,
or put you pipelines to noncaching :-)
>
>
> IMHO POSTing stuff to a source does not really belong to a
> include transformer, but rather to some other type of transformer...
A PostTransformer :-)
Ard
>
> Vadim
>
>
Re: Deprecation of CInclude transformer
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Ard Schrijvers wrote:
<snip/>
> Now, the IncludeTransformer adds the validity object of the included
> sources to the validity object of the calling pipeline (pfffffff....).
> Let's have an example:
>
> someXML.xml :
>
> <doc>
> <include:include src="cocoon://fetchSnippet"/>
> </doc>
>
> <map:match pattern="main">
> <map:generate src="someXML.xml"/>
> <map:transform type="include"/>
> <map:serialize/>
> </map:match
>
> <map:match pattern="fetchSnippet">
> <map:generate src="included.xml"/>
> <map:serialize/>
> </map:match>
>
> included.xml : <includethis/>
>
> So, when calling /main, I will get
>
> <doc>
> <includethis>
> </doc>
>
> and the validity object of the "fetchSnippet" pipeline is added to the
> validity of the "main" pipeline. So, the second call for /main, does the
> setup for the "main" pipeline, and computes a cachekey....and finds a
> hit in the cache with a valid cached object, thus returns the cached
> result. NOTE: the second pipeline is never used in the second call. Not
> even the setup!
>
> Suppose I change now the second matcher the src from included.xml to
> included2.xml, and call /main again....Well, my cachekey of the first
> pipeline is unchanged, as well as the validity object (the validity
> object is with respect to included.xml which did *not* change). So, I
> still get the same cached result. Now, when I open "included.xml",
> change something and save, then the validity obj of the first pipe is
> invalid, so the result is re-computed, fetching included2.xml!!
Indeed. I think this is a failure on CocoonSource part. CocoonSource content
depends on the sitemap.xmap, but CocoonSource validity does not include validity
of the sitemap. As a result, even if sitemap is changed, CocoonSource validity
stays valid.
I'm not sure though if I want to have this bug fixed - it would add more overhead...
> Also, if the second pipeline /fetchSnippet start with a map:select
> first, the things are getting even complexer (the little catch)....if
> somebody wants to know about it I can elaborate.
Fire away! ;-)
> Anyway, why use the IncludeTransformer if you have this 'strange'
> behavior....Well, it is just way much faster than the
> CIncludeTransformer, so if you have pipelines with 10-20 includes (which
> we have in some projects) I want a cached result withing a couplt of ms.
> Do realize, that setting up a pipeline, compute the compound cachekey of
> a generators and transformers, check the cache, check the validity
> object, do take time, which you do not have with the includetransformer
> (it only returns the validity obj). OTOH, it really only makes sense to
> use it when you have a lot of traffic, or when pages have to load fast.
BTW IncludeTransformer operation can not be changed easily. If you try to
resolve each included URI, to check its new key and validity, you'd also have to
parse (or store somewhere!) included parts in order to resolve nested includes -
if recursive inclusion is switched on.
> Hope things are a little clear. Sorry I cannot say to much about the
> CIncludeTransformer...I just do not know enough about it. I just know it
> is much slower (I just looked at the code and see it uses a
> cachingSession with an expires...i do not like it that much i think,
> though it can be used for much more than the IncludeTransformer)
IMHO POSTing stuff to a source does not really belong to a include transformer,
but rather to some other type of transformer...
Vadim
> Regards Ard
>
>> BTW. I forgot to say that I would like to deprecate CInclude
>> in 2.2 only.
>>
>> --
>> Grzegorz Kossakowski
RE: Deprecation of CInclude transformer
Posted by Ard Schrijvers <a....@hippo.nl>.
> AFAIU, the
> IncludeTransformer can only be cached by defining some
Above is ofcourse a typo...I was talking about the CIncludeTransformer
which can only be cached by some expires...
> expires. Clearly, you cannot really know how to set this, or
> if you fetch an external http source, just give it some
> heuristic value.
>
RE: Deprecation of CInclude transformer
Posted by Ard Schrijvers <a....@hippo.nl>.
Hello,
> Grzegorz Kossakowski wrote:
> Could you elaborate on the known differences between CInclude
> and Include transformers? Was it discussed somewhere?
I am not sure if it was ever discussed somewhere, but I can elaborate a
little on it ( though it has been a while, so I might be a little off at
some places... though understanding of this IncludeTransformer gives you
a very good insight in Cocoon's caching mechanism. ).
First of all, I think there are different useages of both transformers,
where I must admit I never used the CInclude ( I know you can also POST
with it, append parameters, etc etc ). With the includeTransformer, all
you do is including a src which can use any of the defined
source-factories in you cocoon.xconf, for example
<include:include src="cocoon://some/pipeline"/> or
<include:include src="file://some/fs/file"/>
We at Hippo *always* used the IncludeTransformer, because it is way
better cacheable (and blistering fast ones included....but ofcourse,
there is a very subtle catch which the author didn't realize i
think...or just did not tell :-) .), as opposed to the CInclude
transformer. AFAIU, the IncludeTransformer can only be cached by
defining some expires. Clearly, you cannot really know how to set this,
or if you fetch an external http source, just give it some heuristic
value.
Now, the IncludeTransformer adds the validity object of the included
sources to the validity object of the calling pipeline (pfffffff....).
Let's have an example:
someXML.xml :
<doc>
<include:include src="cocoon://fetchSnippet"/>
</doc>
<map:match pattern="main">
<map:generate src="someXML.xml"/>
<map:transform type="include"/>
<map:serialize/>
</map:match
<map:match pattern="fetchSnippet">
<map:generate src="included.xml"/>
<map:serialize/>
</map:match>
included.xml : <includethis/>
So, when calling /main, I will get
<doc>
<includethis>
</doc>
and the validity object of the "fetchSnippet" pipeline is added to the
validity of the "main" pipeline. So, the second call for /main, does the
setup for the "main" pipeline, and computes a cachekey....and finds a
hit in the cache with a valid cached object, thus returns the cached
result. NOTE: the second pipeline is never used in the second call. Not
even the setup!
Suppose I change now the second matcher the src from included.xml to
included2.xml, and call /main again....Well, my cachekey of the first
pipeline is unchanged, as well as the validity object (the validity
object is with respect to included.xml which did *not* change). So, I
still get the same cached result. Now, when I open "included.xml",
change something and save, then the validity obj of the first pipe is
invalid, so the result is re-computed, fetching included2.xml!!
Also, if the second pipeline /fetchSnippet start with a map:select
first, the things are getting even complexer (the little catch)....if
somebody wants to know about it I can elaborate.
Anyway, why use the IncludeTransformer if you have this 'strange'
behavior....Well, it is just way much faster than the
CIncludeTransformer, so if you have pipelines with 10-20 includes (which
we have in some projects) I want a cached result withing a couplt of ms.
Do realize, that setting up a pipeline, compute the compound cachekey of
a generators and transformers, check the cache, check the validity
object, do take time, which you do not have with the includetransformer
(it only returns the validity obj). OTOH, it really only makes sense to
use it when you have a lot of traffic, or when pages have to load fast.
Hope things are a little clear. Sorry I cannot say to much about the
CIncludeTransformer...I just do not know enough about it. I just know it
is much slower (I just looked at the code and see it uses a
cachingSession with an expires...i do not like it that much i think,
though it can be used for much more than the IncludeTransformer)
Regards Ard
>
> BTW. I forgot to say that I would like to deprecate CInclude
> in 2.2 only.
>
> --
> Grzegorz Kossakowski
>
Re: Deprecation of CInclude transformer
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Ralph Goers wrote:
> I just got a response from my Engineer. Here is what he said "I never
> got the xinclude to work. I tried doing a search and replace as well as
> hacking cocoon.xconf to replace the cinclude config to utilize the
> xinclude stuff. All I got were blanks, but no errors. But I did not turn
> on debugging."
Xinclude has different namespace and syntax, so it is not a substitute for
CInclude/Include.
OTH CInclude and Include share same namespace but differ in source resolution,
so if there are relative URLs, it is not a drop-in replacement either. But if
your URLs are all absolute (like cocoon: URLs) then it should work.
Vadim
> Ralph
>
> Ralph Goers wrote:
>>
>>
>> Grzegorz Kossakowski wrote:
>>> Could you elaborate on the known differences between CInclude and
>>> Include transformers? Was it discussed somewhere?
>> http://cocoon.apache.org/2.1/apidocs/org/apache/cocoon/transformation/IncludeTransformer.html
>> documents how it is different from CInclude and XInclude.
>>>
>>> BTW. I forgot to say that I would like to deprecate CInclude in 2.2
>>> only.
>> Sure.
Re: Deprecation of CInclude transformer
Posted by Ralph Goers <Ra...@dslextreme.com>.
I just got a response from my Engineer. Here is what he said "I never
got the xinclude to work. I tried doing a search and replace as well as
hacking cocoon.xconf to replace the cinclude config to utilize the
xinclude stuff. All I got were blanks, but no errors. But I did not turn
on debugging."
Ralph
Ralph Goers wrote:
>
>
> Grzegorz Kossakowski wrote:
>> Could you elaborate on the known differences between CInclude and
>> Include transformers? Was it discussed somewhere?
> http://cocoon.apache.org/2.1/apidocs/org/apache/cocoon/transformation/IncludeTransformer.html
> documents how it is different from CInclude and XInclude.
>>
>> BTW. I forgot to say that I would like to deprecate CInclude in 2.2
>> only.
> Sure.
Re: Deprecation of CInclude transformer
Posted by Ralph Goers <Ra...@dslextreme.com>.
Grzegorz Kossakowski wrote:
> Could you elaborate on the known differences between CInclude and
> Include transformers? Was it discussed somewhere?
http://cocoon.apache.org/2.1/apidocs/org/apache/cocoon/transformation/IncludeTransformer.html
documents how it is different from CInclude and XInclude.
>
> BTW. I forgot to say that I would like to deprecate CInclude in 2.2 only.
Sure.
Re: Deprecation of CInclude transformer
Posted by Grzegorz Kossakowski <gr...@tuffmail.com>.
Hello Ralph,
Thanks for your response.
Ralph Goers pisze:
> We are using CInclude transformer extensively in our product. I asked my
> colleague to test the IncludeTransformer a while ago as it should
> perform better but I haven't heard back from him. There is one minor
> issue that the IncludeTransformer resolves relative URIs relative to the
> source document location while the CIncludeTransformer does it
> differently. Hopefully he can deal with that.
Could you elaborate on the known differences between CInclude and
Include transformers? Was it discussed somewhere?
BTW. I forgot to say that I would like to deprecate CInclude in 2.2 only.
--
Grzegorz Kossakowski
Re: Deprecation of CInclude transformer
Posted by Ralph Goers <Ra...@dslextreme.com>.
We are using CInclude transformer extensively in our product. I asked my
colleague to test the IncludeTransformer a while ago as it should
perform better but I haven't heard back from him. There is one minor
issue that the IncludeTransformer resolves relative URIs relative to the
source document location while the CIncludeTransformer does it
differently. Hopefully he can deal with that.
Ralph
Grzegorz Kossakowski wrote:
> Hello
>
> Basing on Vadim's explanation[1]:
> "cinclude is old code, ready to be deprecated. include is new code, with new
> features (like parallel includes), and xinclude is based on a standard. If you
> can add xinclude into include transformer, feel free to deprecate xinclude."
>
> I would like to hear your opinions on deprecating CInclude transformer. The arguments for
> deprecating this transformer are pretty obvious:
> 1. There are three different transformers for the same purpose, we should remove some of them
> 2. There is a transformer (Include) that is meant to replace old, CInclude transformer. So there is
> a replacement for the stuff I would like to deprecate.
>
> Anyone would object if I was going to start a vote?
>
> [1] http://article.gmane.org/gmane.text.xml.cocoon.devel/75885
>
>