You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Pier Fumagalli <pi...@betaversion.org> on 2004/08/24 19:58:40 UTC
Nesting/Aggregating generator...
Heya folks...
I just finished writing a -some-sort-of- FileGenerator + CInclude
aggregator, all in one, with caching support...
My problem lies in the fact that we have some files which gets
transformed to generate some CInclude tags, that include content, and
blablabla...
Problem is that I can't cache the output of this thing, as CInclude
doesn't do that "nicely" (it doesn't have all the variables, as it
doesn't know whether its input has changed or not).
Basically what I done is that I took the two, and merged them
together...
A generator always knows its input (its a file) and can verify whether
it's valid, if it is, then the "<include src=...>" declaration in there
will not be changed, so it can also verify whether its included files
are valid or not...
And so on recursively...
Basically, it's a <map:aggregate> but the aggregation is driven by
what's in the source (and recursively, in all sources so included).
Is there any interest in this? Shall I add?
Can be handy...
Pier
Re: Nesting/Aggregating generator...
Posted by Bertrand Delacretaz <bd...@apache.org>.
Le 24 août 04, à 19:58, Pier Fumagalli a écrit :
> Is there any interest in this?
Yes, sounds like a useful tool!
> Shall I add?
See above ;-)
-Bertrand
Re: [OT] Subversion --migrate option missing (Was: Re: Nesting/Aggregating
generator...)
Posted by Giacomo Pati <gi...@apache.org>.
On Wed, 25 Aug 2004, Vadim Gritsenko wrote:
> Pier Fumagalli wrote:
>
>> On 25 Aug 2004, at 02:17, Vadim Gritsenko wrote:
>>
>>> Pier Fumagalli wrote:
>>>
>>>> On 24 Aug 2004, at 22:28, Pier Fumagalli wrote:
>>>> Ahhh.... FOR CRYING OUT LOUD! :-( I don't have access to Subversion
>>>> :-(
>>>>> Commit failed (details follow):
>>>>> MKACTIVITY of
>>>> /repos/asf/!svn/act/ed3896a8-6ee2-0310-b4ce-bc301ee366a2: 403
>>>> Forbidden (http://svn.apache.org)
>>>
>>>
>>>> -)
>>>
>>> Please use https://, and you can do "svn sw --migrate" (see [1]).
>>>
>>> [1] http://wiki.apache.org/cocoon/SubversionMigration
>>
>>
>> Took me 3 hours (as I had to recompile subversion, haha!)....
>>
>> I don't have the "--migrate" option in my subversion client... Where does
>> it come from? What does it do? Shall we update the wiki?
>
> Hm, mine does not have it either. The goal is to switch existing checkout
> from http:// to https://. May be --relocate is what you need? Help says...
>
> 2. switch --relocate FROM TO [PATH...]
>
> 2. Rewrite working copy URL metadata to reflect a syntactic change only.
> This is used when repository's root URL changes (such as a schema
> or hostname change) but your working copy still reflects the same
> directory within the same repository.
>
>
> And here we have only schema change.
Its exactly that:
svn switch --relocate http://svn... https://svn...
Will change the url in the metadata (.svn/... dirs)
>
>
> Alternatively, you can just make fresh checkout from https:// and move your
> changes manually into this new checkout.
>
> Vadim
>
>
>
--
Giacomo Pati
Otego AG, Switzerland - http://www.otego.com
Orixo, the XML business alliance - http://www.orixo.com
Re: [OT] Subversion --migrate option missing (Was: Re: Nesting/Aggregating
generator...)
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Pier Fumagalli wrote:
> On 25 Aug 2004, at 02:17, Vadim Gritsenko wrote:
>
>> Pier Fumagalli wrote:
>>
>>> On 24 Aug 2004, at 22:28, Pier Fumagalli wrote:
>>> Ahhh.... FOR CRYING OUT LOUD! :-( I don't have access to Subversion :-(
>>> svn: Commit failed (details follow):
>>> svn: MKACTIVITY of
>>> /repos/asf/!svn/act/ed3896a8-6ee2-0310-b4ce-bc301ee366a2: 403
>>> Forbidden (http://svn.apache.org)
>>
>>
>> :-)
>>
>> Please use https://, and you can do "svn sw --migrate" (see [1]).
>>
>> [1] http://wiki.apache.org/cocoon/SubversionMigration
>
>
> Took me 3 hours (as I had to recompile subversion, haha!)....
>
> I don't have the "--migrate" option in my subversion client... Where
> does it come from? What does it do? Shall we update the wiki?
Hm, mine does not have it either. The goal is to switch existing
checkout from http:// to https://. May be --relocate is what you need?
Help says...
2. switch --relocate FROM TO [PATH...]
2. Rewrite working copy URL metadata to reflect a syntactic change only.
This is used when repository's root URL changes (such as a schema
or hostname change) but your working copy still reflects the same
directory within the same repository.
And here we have only schema change.
Alternatively, you can just make fresh checkout from https:// and move
your changes manually into this new checkout.
Vadim
[OT] Subversion --migrate option missing (Was: Re: Nesting/Aggregating generator...)
Posted by Pier Fumagalli <pi...@betaversion.org>.
On 25 Aug 2004, at 02:17, Vadim Gritsenko wrote:
> Pier Fumagalli wrote:
>
>> On 24 Aug 2004, at 22:28, Pier Fumagalli wrote:
>> Ahhh.... FOR CRYING OUT LOUD! :-( I don't have access to Subversion
>> :-(
>> svn: Commit failed (details follow):
>> svn: MKACTIVITY of
>> /repos/asf/!svn/act/ed3896a8-6ee2-0310-b4ce-bc301ee366a2: 403
>> Forbidden (http://svn.apache.org)
>
> :-)
>
> Please use https://, and you can do "svn sw --migrate" (see [1]).
>
> [1] http://wiki.apache.org/cocoon/SubversionMigration
Took me 3 hours (as I had to recompile subversion, haha!)....
I don't have the "--migrate" option in my subversion client... Where
does it come from? What does it do? Shall we update the wiki?
Pier
Re: Nesting/Aggregating generator...
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Pier Fumagalli wrote:
> On 24 Aug 2004, at 22:28, Pier Fumagalli wrote:
>
> Ahhh.... FOR CRYING OUT LOUD! :-( I don't have access to Subversion :-(
>
> svn: Commit failed (details follow):
> svn: MKACTIVITY of
> /repos/asf/!svn/act/ed3896a8-6ee2-0310-b4ce-bc301ee366a2: 403 Forbidden
> (http://svn.apache.org)
:-)
Please use https://, and you can do "svn sw --migrate" (see [1]).
> Can someone post this? If the validity is not "closed", it will always
> say it's invalid no matter what, and therefore the caching won't work...
>
> Also, anyone against me patching this one to allow the recursive
> behavior (upon configuration, obviously) of includes within includes?
Hm, how would that work? :)
Vadim
[1] http://wiki.apache.org/cocoon/SubversionMigration
Re: Nesting/Aggregating generator...
Posted by Pier Fumagalli <pi...@betaversion.org>.
On 24 Aug 2004, at 22:28, Pier Fumagalli wrote:
> On 24 Aug 2004, at 21:48, Pier Fumagalli wrote:
>> On 24 Aug 2004, at 19:07, Vadim Gritsenko wrote:
>>>
>>> How it will differ from [1]? IncludeTransformer was added recently
>>> exactly to address this problem with CIncludeTransformer. See also
>>> bug #30356.
>>>
>>> Vadim
>>>
>>> [1]
>>> http://svn.apache.org/repos/asf/cocoon/trunk/src/blocks/scratchpad/
>>> java/org/apache/cocoon/transformation/IncludeTransformer.java
>>
>> Well, at a first glance, I'd say that the IncludeTransformer is
>> breaking up quite a lot of rules regarding caching...
>
> Disregard entirely... I'm f**king stupid... See the
> "URLEncodingTransformer" which is by all means cachable.
>
> Nope, then... The union of FileGenerator + IncludeTransformer or
> PiersIncludeGenerator don't have anything different, apart from a
> little subtlety regarding the recursive-inclusion behavior (if one of
> the included sources contains the inclusion element, it'll be
> processed recursively).
Ahhh.... FOR CRYING OUT LOUD! :-( I don't have access to Subversion :-(
svn: Commit failed (details follow):
svn: MKACTIVITY of
/repos/asf/!svn/act/ed3896a8-6ee2-0310-b4ce-bc301ee366a2: 403 Forbidden
(http://svn.apache.org)
Can someone post this? If the validity is not "closed", it will always
say it's invalid no matter what, and therefore the caching won't
work...
Also, anyone against me patching this one to allow the recursive
behavior (upon configuration, obviously) of includes within includes?
Pier
PS. How do I get access to SVN?
PPS. Da patch below!
Index: IncludeTransformer.java
===================================================================
--- IncludeTransformer.java (revision 36825)
+++ IncludeTransformer.java (working copy)
@@ -81,6 +81,20 @@
m_validity = null;
}
+ public void startDocument()
+ throws SAXException {
+ // Make sure that we have a validity while processing
+ this.getValidity();
+ super.startDocument();
+ }
+
+ public void endDocument()
+ throws SAXException {
+ // Make sure that the validity is "closed" at the end
+ this.m_validity.close();
+ super.endDocument();
+ }
+
public void startElement(String uri, String localName, String
qName, Attributes atts)
throws SAXException {
if (NS_URI.equals(uri)) {
@@ -89,9 +103,7 @@
Source source = null;
try {
source = m_resolver.resolveURI(src);
- if (m_validity != null) {
- m_validity.addSource(source);
- }
+ m_validity.addSource(source);
SourceUtil.toSAX(m_manager, source, "text/xml",
new
IncludeXMLConsumer(super.contentHandler));
}
Re: Nesting/Aggregating generator...
Posted by Pier Fumagalli <pi...@betaversion.org>.
On 24 Aug 2004, at 21:48, Pier Fumagalli wrote:
> On 24 Aug 2004, at 19:07, Vadim Gritsenko wrote:
>>
>> How it will differ from [1]? IncludeTransformer was added recently
>> exactly to address this problem with CIncludeTransformer. See also
>> bug #30356.
>>
>> Vadim
>>
>> [1]
>> http://svn.apache.org/repos/asf/cocoon/trunk/src/blocks/scratchpad/
>> java/org/apache/cocoon/transformation/IncludeTransformer.java
>
> Well, at a first glance, I'd say that the IncludeTransformer is
> breaking up quite a lot of rules regarding caching...
Disregard entirely... I'm f**king stupid... See the
"URLEncodingTransformer" which is by all means cachable.
Nope, then... The union of FileGenerator + IncludeTransformer or
PiersIncludeGenerator don't have anything different, apart from a
little subtlety regarding the recursive-inclusion behavior (if one of
the included sources contains the inclusion element, it'll be processed
recursively).
Pier
Re: Nesting/Aggregating generator...
Posted by Pier Fumagalli <pi...@betaversion.org>.
On 24 Aug 2004, at 19:07, Vadim Gritsenko wrote:
>
> How it will differ from [1]? IncludeTransformer was added recently
> exactly to address this problem with CIncludeTransformer. See also bug
> #30356.
>
> Vadim
>
> [1]
> http://svn.apache.org/repos/asf/cocoon/trunk/src/blocks/scratchpad/
> java/org/apache/cocoon/transformation/IncludeTransformer.java
Well, at a first glance, I'd say that the IncludeTransformer is
breaking up quite a lot of rules regarding caching...
getKey always returns "IncludeTransformer" which I don't know exactly
what kind of behaviour might trigger in the backend cache
implementation (I'm not saying "it doesn't work", I'm saying it _might_
not work).
If I have a pipeline like the following:
<map:pipeline>
<map:match pattern="*"/>
<map:generate src="{1}.xml"/>
<map:transform type="include"/>
<map:serialize/>
</map:match>
</map:pipeline>
When I do a request for "foo" the key for the cache validity produced
by the generator will be "file://foo.xml" (for example), if I request
"bar" it'll be "file://bar.xml"...
The transformer always returns "IncludeTransformer", no question
asked...
So, maybe with the current cache implementation it works, but from what
I can see from the API docs, there might be a lot of cases in which
you're going to get some pretty weird behaviors out of it, especially
when your cache grows _a_lot_ (we have a diversity in URLs alone of
approximately 150.000 entries, without counting the internal URLs,
cocoon:/....) and you need to start relaying on disk to
serialize/de-serialize caches and validities.
Pier
Re: Nesting/Aggregating generator...
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Pier Fumagalli wrote:
>
> I just finished writing a -some-sort-of- FileGenerator + CInclude
> aggregator, all in one, with caching support...
>
> My problem lies in the fact that we have some files which gets
> transformed to generate some CInclude tags, that include content, and
> blablabla...
>
> Problem is that I can't cache the output of this thing, as CInclude
> doesn't do that "nicely" (it doesn't have all the variables, as it
> doesn't know whether its input has changed or not).
>
> Basically what I done is that I took the two, and merged them together...
>
> A generator always knows its input (its a file) and can verify whether
> it's valid, if it is, then the "<include src=...>" declaration in there
> will not be changed, so it can also verify whether its included files
> are valid or not...
>
> And so on recursively...
>
> Basically, it's a <map:aggregate> but the aggregation is driven by
> what's in the source (and recursively, in all sources so included).
>
> Is there any interest in this? Shall I add?
How it will differ from [1]? IncludeTransformer was added recently
exactly to address this problem with CIncludeTransformer. See also bug
#30356.
Vadim
[1]
http://svn.apache.org/repos/asf/cocoon/trunk/src/blocks/scratchpad/java/org/apache/cocoon/transformation/IncludeTransformer.java