You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by MAISONOBE Luc <lu...@c-s.fr> on 2003/06/25 08:28:11 UTC
residuals of MIME type bug ?
Hello,
I have updated my forrest source tree yesterday evening (european time)
including the new cocoon jar.
This has solved the file.extnull problem I had encountered like everyone
else, and also partly solved my src/content/file.xml problems (those
files were parsed and slightly modified despite they are not in the
xdocs sub-directory). The files are now correctly copied untouched.
The remaining problems I see are :
- a single java source file that should also be copied as is has
its name changed from EllipseArc.java to EllipseArc.java.txt
(before the cocoon change, it was renamed with the .text suffix
rather than .txt so I think this is still related to the same
bug)
- a compressed PostScript file is copied twice, once as is and
another time in decompressed form (with the .gz suffix removed)
- spurious error messages appear during site generation:
Despite the files are copied near the beginning of forrest run,
I get the following error messages later on (and the files are
listed in the brokenlist.txt file). I noticed also that one of
my xml file did generate an error message and was not listed. I
did not understand why this one was different from the other
ones.
------------------------------------------------------------------
...lots of stuff omitted ...
* [0] skin/images/valid-html401.png
* [41] downloads.html
* [0]
-> [broken page] software/rkcheck/HighamHall.xml <-
/home/luc/spaceroots.org/forrest/build/tmp/context/content/xd
ocs/software/rkcheck/HighamHall.xml (No such file or director
y)
...lots of stuff omitted ...
------------------------------------------------------------------
It seems as if due to the .xml suffix, some part of the
process expect the file to really live in the xdocs directory,
and logically doesn't find them here. I suspect my file names
are recognized by two different rules, one based on their
directory (not in xdocs), and another one based on their suffix
(*.xml). The first rule working correctly for these file types now,
and the other rule failing. If this is the case, a simple correction
would be to extend the pattern to something like */xdocs/*.xml
or similar. Does it make sense ?
I can try to look at this problem by myself, could you advise me were to
start (sitemap.xmap only ?). I don't think I can handle the first two
problems though.
Luc
Re: residuals of MIME type bug ?
Posted by Jeff Turner <je...@apache.org>.
On Wed, Jun 25, 2003 at 08:28:11AM +0200, MAISONOBE Luc wrote:
> Hello,
>
> I have updated my forrest source tree yesterday evening (european time)
> including the new cocoon jar.
>
> This has solved the file.extnull problem I had encountered like everyone
> else, and also partly solved my src/content/file.xml problems (those
> files were parsed and slightly modified despite they are not in the
> xdocs sub-directory). The files are now correctly copied untouched.
>
> The remaining problems I see are :
>
> - a single java source file that should also be copied as is has
> its name changed from EllipseArc.java to EllipseArc.java.txt
> (before the cocoon change, it was renamed with the .text suffix
> rather than .txt so I think this is still related to the same
> bug)
I can reproduce the problem, but haven't had a chance to debug. Cocoon's
command-line processor will 'correct' the extension for the few MIME
types it knows about, and not touch the extension for MIME types it
doesn't know. The problem is that *.java files are being served by the
sitemap (raw.xmap) with MIME type text/plain, hence the .txt extension.
I think the quick fix would be to override raw.xmap (copy from
$FORREST/context/raw.xmap to src/documentation/) and declare a matcher
for **.java, which sets the MIME type to text/plain+java or something.
The other possible solution is to extract the mime.types file from the
Cocoon jar, and comment out the text/plain entry:
[xml-forrest ~/build/dist/shbat/WEB-INF/classes]$ jar xvf
../lib/cocoon-20030622.jar org/apache/cocoon/util/mime.types
> - a compressed PostScript file is copied twice, once as is and
> another time in decompressed form (with the .gz suffix removed)
Bizarre. Does the *.xml file link to the compressed or uncompressed
version? If you run 'forrest run', does Jetty serve the content
compressed or uncompressed?
> - spurious error messages appear during site generation:
> Despite the files are copied near the beginning of forrest run,
> I get the following error messages later on (and the files are
> listed in the brokenlist.txt file). I noticed also that one of
> my xml file did generate an error message and was not listed. I
> did not understand why this one was different from the other
> ones.
>
> ------------------------------------------------------------------
>
> ...lots of stuff omitted ...
>
> * [0] skin/images/valid-html401.png
> * [41] downloads.html
> * [0]
> -> [broken page] software/rkcheck/HighamHall.xml <-
>
> /home/luc/spaceroots.org/forrest/build/tmp/context/content/xd
> ocs/software/rkcheck/HighamHall.xml (No such file or director
> y)
>
> ------------------------------------------------------------------
>
> It seems as if due to the .xml suffix, some part of the
> process expect the file to really live in the xdocs directory,
> and logically doesn't find them here. I suspect my file names
> are recognized by two different rules, one based on their
> directory (not in xdocs), and another one based on their suffix
> (*.xml). The first rule working correctly for these file types now,
> and the other rule failing. If this is the case, a simple correction
> would be to extend the pattern to something like */xdocs/*.xml
> or similar. Does it make sense ?
Ah yes, that's probably it. If you look in sitemap.xmap, you'll see the
source matchers (**.xml, reading from content/xdocs/*.xml) are located
before the serve-raw-content matcher:
251 <map:pipeline internal-only="false">
252
253 <!-- ============================================================ -->
254 <!-- OUTPUT FORMATS -->
255 <!-- Serves content directly to the user -->
256 <!-- +==========================================================+ -->
257 <map:match type="regexp" pattern="^.+$">
258 <map:select type="exists">
259 <map:when test="content/{0}">
260 <map:mount uri-prefix="" src="raw.xmap" check-reload="yes" />
261 </map:when>
262 </map:select>
263 </map:match>
Moving that block above the source matchers might help:
130 <map:pipeline internal-only="false">
131
--->
132
133 <!-- ============================================================ -->
134 <!-- SOURCE FORMATS -->
135 <!-- Raw XML sources, typically doc-v11 format -->
136 <!-- ============================================================ -->
137
138 <map:match pattern="changes.xml">
139 <map:mount uri-prefix="" src="status.xmap" check-reload="yes" />
140 </map:match>
I'll have a look tomorrow.
--Jeff
> I can try to look at this problem by myself, could you advise me were to
> start (sitemap.xmap only ?). I don't think I can handle the first two
> problems though.
>
> Luc
>
>
Re: residuals of MIME type bug ?
Posted by Vadim Gritsenko <va...@verizon.net>.
Steven Noels wrote:
> On 26/06/2003 12:33 yachting heritage wrote:
>
>> how could we avoid to received your bloody mail?
>
>
> sorry, didn't meant to moderate that one through :-|
>
> trying to remotely unsubscribe this friendly person...
Thank you for unsubscribing him from our bloody lists!!!
:)))
Vadim
Re: residuals of MIME type bug ?
Posted by Steven Noels <st...@outerthought.org>.
On 26/06/2003 12:33 yachting heritage wrote:
> how could we avoid to received your bloody mail?
sorry, didn't meant to moderate that one through :-|
trying to remotely unsubscribe this friendly person...
</Steven>
--
Steven Noels http://outerthought.org/
Outerthought - Open Source, Java & XML Competence Support Center
Read my weblog at http://blogs.cocoondev.org/stevenn/
stevenn at outerthought.org stevenn at apache.org
Re: residuals of MIME type bug ?
Posted by Steven Noels <st...@outerthought.org>.
On 26/06/2003 12:33 yachting heritage wrote:
> how could we avoid to received your bloody mail?
sorry, didn't meant to moderate that one through :-|
trying to remotely unsubscribe this friendly person...
</Steven>
--
Steven Noels http://outerthought.org/
Outerthought - Open Source, Java & XML Competence Support Center
Read my weblog at http://blogs.cocoondev.org/stevenn/
stevenn at outerthought.org stevenn at apache.org
Re: residuals of MIME type bug ?
Posted by yachting heritage <co...@yachting-heritage.com>.
how could we avoid to received your bloody mail?
----- Original Message -----
From: "Jeff Turner" <je...@apache.org>
To: <fo...@xml.apache.org>
Cc: <co...@xml.apache.org>
Sent: Thursday, June 26, 2003 10:18 AM
Subject: Re: residuals of MIME type bug ?
> (moving to cocoon-dev..)
>
> On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
> > There are quite a lot of new features in the Cocoon CLI that Forrest
> > isn't using, for example the option to switch off mime-type checking,
> > and to only scan pages once (i.e. not using the link-view) to follow
> > links.
>
> But I like the link-views! ;) It's one of those design elegancies that
> makes Cocoon unique. Adding a don't-crawl-these-links option to the new
> CLI may solve the same problem, but IMHO it's a hack in comparison.
>
> The argument against link-views is that it's slow: two requests instead
> of one. Isn't the correct solution to fix caching so that 99% of the
> processing between foo.html and foo.html?cocoon-view=links is shared? If
> caching worked properly, why would requesting the link view take much
> more time?
>
>
> --Jeff
>
> ...
> > Regards, Upayavira
> >
>
>
Re: residuals of MIME type bug ?
Posted by Jeff Turner <je...@apache.org>.
On Thu, Jun 26, 2003 at 05:59:47PM +0200, Nicola Ken Barozzi wrote:
>
> Vadim Gritsenko wrote, On 26/06/2003 14.38:
> >Jeff Turner wrote:
> >
> >>(moving to cocoon-dev..)
> >>
> >>On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
> >>
> >>
> >>>There are quite a lot of new features in the Cocoon CLI that Forrest
> >>>isn't using, for example the option to switch off mime-type checking,
> >>>and to only scan pages once (i.e. not using the link-view) to follow
> >>>links.
> >>>
> >>
> >>
> >>But I like the link-views! ;) It's one of those design elegancies that
> >>makes Cocoon unique. Adding a don't-crawl-these-links option to the new
> >>CLI may solve the same problem, but IMHO it's a hack in comparison.
>
> IMHO it's the link-view that's a hack. How it's done now, it's
> completely mixed between external and internal needs.
What is an external/internal need?
> I mean, what we have is an environment that to work has to have a
> certain view specified in the sitemap...
We could have a built-in links view if one isn't defined in the sitemap.
> >>Isn't the correct solution to fix caching so that 99% of the
> >>processing between foo.html and foo.html?cocoon-view=links is shared?
>
> It's not so. The links are in the content view, and after that there is
> a lot of processing.
Well there you go: yet another reason for _not_ hardcoding the link
extraction just before the serializer :) By making the links view
explicit in the sitemap, I can change:
<map:view name="links" from-position="last">
to:
<map:view name="links" from-position="content">
> >>If
> >>caching worked properly, why would requesting the link view take much
> >>more time?
>
> Because they do not share 99% of the same stuff.
In the standard links view:
<map:view name="links" from-position="last">
<map:serialize type="links"/>
</map:view>
All processing up to the serializer is shared, if caching works
optimally.
> >Have you tried caching point pipeline; and/or pipeline with caching
> >hints? IIUC, you need to hint sitemap to cache data of the pipeline
> >fragment common to both normal processing and links view.
>
> Thgis alleviates something, but it will never probably get down to the
> speed of the new method.
>
> Please Jeff, take a moment and read all the thread that brought to the
> creation of the new method, it's fully explained and discussed in detail.
Did that thread have any discussion of the _value_ of link views, or what
would be lost by discarding them?
--Jeff
> --
> Nicola Ken Barozzi nicolaken@apache.org
> - verba volant, scripta manent -
> (discussions get forgotten, just code remains)
> ---------------------------------------------------------------------
Re: residuals of MIME type bug ?
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Vadim Gritsenko wrote, On 26/06/2003 14.38:
> Jeff Turner wrote:
>
>> (moving to cocoon-dev..)
>>
>> On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
>>
>>
>>> There are quite a lot of new features in the Cocoon CLI that Forrest
>>> isn't using, for example the option to switch off mime-type checking,
>>> and to only scan pages once (i.e. not using the link-view) to follow
>>> links.
>>>
>>
>>
>> But I like the link-views! ;) It's one of those design elegancies that
>> makes Cocoon unique. Adding a don't-crawl-these-links option to the new
>> CLI may solve the same problem, but IMHO it's a hack in comparison.
IMHO it's the link-view that's a hack. How it's done now, it's
completely mixed between external and internal needs.
I mean, what we have is an environment that to work has to have a
certain view specified in the sitemap...
>> The argument against link-views is that it's slow: two requests instead
>> of one.
Actually for the original method it's three.
>> Isn't the correct solution to fix caching so that 99% of the
>> processing between foo.html and foo.html?cocoon-view=links is shared?
It's not so. The links are in the content view, and after that there is
a lot of processing.
>> If
>> caching worked properly, why would requesting the link view take much
>> more time?
Because they do not share 99% of the same stuff.
> Have you tried caching point pipeline; and/or pipeline with caching
> hints? IIUC, you need to hint sitemap to cache data of the pipeline
> fragment common to both normal processing and links view.
Thgis alleviates something, but it will never probably get down to the
speed of the new method.
Please Jeff, take a moment and read all the thread that brought to the
creation of the new method, it's fully explained and discussed in detail.
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: residuals of MIME type bug ?
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Vadim Gritsenko wrote, On 26/06/2003 14.38:
> Jeff Turner wrote:
>
>> (moving to cocoon-dev..)
>>
>> On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
>>
>>
>>> There are quite a lot of new features in the Cocoon CLI that Forrest
>>> isn't using, for example the option to switch off mime-type checking,
>>> and to only scan pages once (i.e. not using the link-view) to follow
>>> links.
>>>
>>
>>
>> But I like the link-views! ;) It's one of those design elegancies that
>> makes Cocoon unique. Adding a don't-crawl-these-links option to the new
>> CLI may solve the same problem, but IMHO it's a hack in comparison.
IMHO it's the link-view that's a hack. How it's done now, it's
completely mixed between external and internal needs.
I mean, what we have is an environment that to work has to have a
certain view specified in the sitemap...
>> The argument against link-views is that it's slow: two requests instead
>> of one.
Actually for the original method it's three.
>> Isn't the correct solution to fix caching so that 99% of the
>> processing between foo.html and foo.html?cocoon-view=links is shared?
It's not so. The links are in the content view, and after that there is
a lot of processing.
>> If
>> caching worked properly, why would requesting the link view take much
>> more time?
Because they do not share 99% of the same stuff.
> Have you tried caching point pipeline; and/or pipeline with caching
> hints? IIUC, you need to hint sitemap to cache data of the pipeline
> fragment common to both normal processing and links view.
Thgis alleviates something, but it will never probably get down to the
speed of the new method.
Please Jeff, take a moment and read all the thread that brought to the
creation of the new method, it's fully explained and discussed in detail.
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: residuals of MIME type bug ?
Posted by Vadim Gritsenko <va...@verizon.net>.
Jeff Turner wrote:
>(moving to cocoon-dev..)
>
>On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
>
>
>>There are quite a lot of new features in the Cocoon CLI that Forrest
>>isn't using, for example the option to switch off mime-type checking,
>>and to only scan pages once (i.e. not using the link-view) to follow
>>links.
>>
>>
>
>But I like the link-views! ;) It's one of those design elegancies that
>makes Cocoon unique. Adding a don't-crawl-these-links option to the new
>CLI may solve the same problem, but IMHO it's a hack in comparison.
>
>The argument against link-views is that it's slow: two requests instead
>of one. Isn't the correct solution to fix caching so that 99% of the
>processing between foo.html and foo.html?cocoon-view=links is shared? If
>caching worked properly, why would requesting the link view take much
>more time?
>
Have you tried caching point pipeline; and/or pipeline with caching
hints? IIUC, you need to hint sitemap to cache data of the pipeline
fragment common to both normal processing and links view.
Vadim
Re: residuals of MIME type bug ?
Posted by yachting heritage <co...@yachting-heritage.com>.
how could we avoid to received your bloody mail?
----- Original Message -----
From: "Jeff Turner" <je...@apache.org>
To: <fo...@xml.apache.org>
Cc: <co...@xml.apache.org>
Sent: Thursday, June 26, 2003 10:18 AM
Subject: Re: residuals of MIME type bug ?
> (moving to cocoon-dev..)
>
> On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
> > There are quite a lot of new features in the Cocoon CLI that Forrest
> > isn't using, for example the option to switch off mime-type checking,
> > and to only scan pages once (i.e. not using the link-view) to follow
> > links.
>
> But I like the link-views! ;) It's one of those design elegancies that
> makes Cocoon unique. Adding a don't-crawl-these-links option to the new
> CLI may solve the same problem, but IMHO it's a hack in comparison.
>
> The argument against link-views is that it's slow: two requests instead
> of one. Isn't the correct solution to fix caching so that 99% of the
> processing between foo.html and foo.html?cocoon-view=links is shared? If
> caching worked properly, why would requesting the link view take much
> more time?
>
>
> --Jeff
>
> ...
> > Regards, Upayavira
> >
>
>
Re: residuals of MIME type bug ?
Posted by Vadim Gritsenko <va...@verizon.net>.
Jeff Turner wrote:
>(moving to cocoon-dev..)
>
>On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
>
>
>>There are quite a lot of new features in the Cocoon CLI that Forrest
>>isn't using, for example the option to switch off mime-type checking,
>>and to only scan pages once (i.e. not using the link-view) to follow
>>links.
>>
>>
>
>But I like the link-views! ;) It's one of those design elegancies that
>makes Cocoon unique. Adding a don't-crawl-these-links option to the new
>CLI may solve the same problem, but IMHO it's a hack in comparison.
>
>The argument against link-views is that it's slow: two requests instead
>of one. Isn't the correct solution to fix caching so that 99% of the
>processing between foo.html and foo.html?cocoon-view=links is shared? If
>caching worked properly, why would requesting the link view take much
>more time?
>
Have you tried caching point pipeline; and/or pipeline with caching
hints? IIUC, you need to hint sitemap to cache data of the pipeline
fragment common to both normal processing and links view.
Vadim
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Upayavira <uv...@upaya.co.uk>.
On 1 Jul 2003 at 14:47, Vadim Gritsenko wrote:
> Jeff Turner wrote:
>
> >I'm not very familiar with the code; is there some cost in keeping
> >the two-pass CLI alive, in the faint hope that caching comes to its
> >rescue one day?
> >
>
> Guys,
>
> Before you implement some approach here... Let me suggest something.
>
> Right now sitemap implementation automatically adds link gatherer to
> the pipeline when it is invoked by CLI. This link gatherer is in fact
> is "hard-coded links view". I suggest to replace this "hard-coded
> links view" a.k.a link gatherer with the "real" links view, BUT attach
> it as a tee to a main pipeline instead of running it as a pipeline by
> itself. As a result, links view "baby" will be used, two-pass "water"
> will be drained, and sitemap syntax will stay the same. Moreover, the
> links view will be still accessible from the outside, meaning that you
> can spider the site using out-of-the-process spiders.
>
> Example:
> Given the pipeline:
> G --> T1 (label="content") --> T2 --> S,
>
> And the links view:
> from-label="content" --> T3 --> LinkSerializer,
>
> The pipeline built for the CLI request should be:
> G --> T1 --> Tee --> T2 --> S --> OutputStream
> \
> --> LinkSerializer --> NullOutputStream
> \
> --> List of links in environment
>
> In one request, you will get:
> * Regular output of the pipeline which will go to the destination
> Source * List of links in the environment which is what link gatherer
> was made for
Splendid. I think that is exactly what I would want to do. We'd then have single(ish)
pass generation with the benefits of link view. And if you just feed directly from the
label into a serializer, it'll be pretty much the same in terms of performance as the
LinkGatherer that we have now.
I would need help implementing this. Are you able to explain how?
There's a lot of pipeline building there that I wouldn't yet know how to do (but I'm
willing to give it a go with guidance).
If we're to use my current approach, we'd add a different serializer at the end of the
second sub-pipe, which would take the links and put them into a specific List in the
ObjectModel. In fact, we could create a LinkGatheringOutputStream that'd be handed
to the LinkSerializer to do that. That would leave most of the complexity simply in
building the pipeline.
Can you guarantee that cocoon.process() will not complete until both sub-pipelines
have completed their work?
I'll take a bit of a look into the pipeline building code (if I can find it) to see what I can
work out.
This approach excites me. With help, I'd like to see if I can make it happen.
Regards, Upayavira
Re: Aspect-based pipelines and link view ( Re: Link view goodness)
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Upayavira wrote, On 30/06/2003 18.18:
> Guys,
>
>
>>The link stuff is a cross-cutting concern. This thread has IMHO shown
>>how aspects can be easily added to the sitemap, and effectively used.
>>Let's see...
...
>>><map:resource name="gather-links" from-position="content">
>>> <!-- Any required link munging -->
>>> <map:transformer type="gather-links"/>
>>></map:resource>
...
>>Seems like simply adding this capability to resources is nice. We
>>could similarly make a link-view that uses the same transformer and a
>>serializer. In this way it could also be compatible with the 3pass
>>method. Hmmm...
>>
>>>Ie, a Resource inserted in each pipeline after the 'content' label.
>>>Rather AOP'ish.
>>
>>Yup. As link gathering is a cross-cutting concern, it also makes sense
>>conceptually.
>
> So you're saying, with a resource that has a 'from-position' attribute, that specifies
> after which label it should be inserted? That makes sense. So you only have to have
> the resource once per sitemap, rather than having to insert it into every pipeline.
Exactly.
As you say, it automatically gets inserted in each place where I would
have to put it manually.
So it's not part of the single pipelines, but it's a common "aspect" of
the system that gets applied.
For example, if I wanted to log all method exits, I could put by hand a
log() call in each method. Or I could factor out this aspect and specify
with a rule where it applies, like saying:
foreach(method.return){
log()
}
The important thing here is how I can tell the system about where to
apply that code. It's important that it can be explained as a common
rule, or else I'll be just moving the single log calls out of the
methods, with no gain.
Here, linking has to be applied to a *group* of pipelines, in a specific
*part* of them.
IE: "for xml pipelines, where the content is done, insert this transformer".
> But - what if the pipeline itself needs modifying to expose links from within PDFs for
> example. The LinkGatheringTransformer I have coded has two modes, one where it
> just hunts for href, src and xlink attributes, and the other that searches for attributes
> in the http://apache.org/cocoon/link-gatherer/1.0 namespace (probably to be used
> with a 'link' prefix). This latter kind is required for gathering links that don't conform to
> the href, src or xlink conventions. Just auto-inserting a link gatherer wouldn't work in
> this case.
It has to be possible to define when one or the other apply.
Examples of the two usages?
...
>>Now we say: "when the view is triggered, start at a label"
>>After it could be: "when the view is triggered, start at position"
>>Instead we need: "when the position is met, check if it has to be
>>triggered".
>>
>>Here is an example that uses this "inverted" AOPish system for views.
>>
>>The following adds two aspects:
>> - an aspect gets called from every content position and gathers
>> links. - the other one gets called from every content position. If
>> the
>>request has a cocoon-view=links, then the links are serialized.
>>
>><map:aspects>
>> <map:aspect type="from-label" test="content">
>> <!-- Any required link munging -->
>> <map:transformer type="gather-links"/>
>> </map:aspect>
>> <map:aspect type="from-label" test="content">
>> <map:action type="request-param">
>> <map:param name="cocoon-view" value="link">
>> <map:serializer type="links"/>
>> </map:action>
>> </map:aspect>
>></map:aspects>
>>
>>This would make it very easy to add security-based checks, logging, or
>>any other stuff.
>>
>><map:aspects>
>> <map:aspect type="pipeline" test="start">
>> <map:action type="check-security"/>
>> </map:aspect>
>> <map:aspect type="pipeline" test="all">
>> <map:transformer type="logger"/>
>> </map:aspect>
>> <map:aspect type="error" test="all">
>> <map:action type="notify-admin"/>
>> </map:aspect>
>></map:aspects>
>>
>>What do others think?
...
>
> I'm afraid you left me completely behind there. I've not really yet understood what
> AOP is, and your ideas go far further than my Cocoon implementation skills currently
> allow.
It's quite easy once you go behind the words...
> I'd quite like to find something that can be implemented reasonably short term, and
> then explore these more far-reaching ideas as time passes (and as the size and
> capacity of my brain increases).
>
> Are you guys interested for the time being in a LinkGatheringTransformer as
> described above? Or is there something not too far away that we can do now to
> gather links?
- Upayavira sanity check System -
- please wait... -
- ... -
- control NicolaKen ...
- in progress...
*** ALERT ALERT ***
*** Highly volatile thoughts ***
*** ALERT ALERT ***
Ok, I got the message ;-)
Thanks for bringing me back to earth, I have the ?slight? ;-) tendency
of flying high.
Let's see what we need to resolve for this case.
We said that the current gatherer has the problem of being fixed.
We have seen that a transformer placed in the right place can be more
configurable.
So, the reasonable solution that comes from this is that if the
user-defined pipeline is caching links, those are used.
If not, Cocoon inserts a gatherer in the right position.
This mixes ease of use with configurability right now.
But then we need amore definitive solution.
Without calling them aspects, let's call them inserts for now.
They are rules that make Cocoon insert resources or pipelines give
particular rules.
For example, let's say that we want Cocoon to add a
LinkGatheringTransformer at the end of each pipeline.
<map:insert type="index" location="last">
<map:transformer type="gather-links"/>
</map:insert>
Let's say we want to add it at the beginning instead:
<map:insert type="index" location="last">
<map:transformer type="gather-links"/>
</map:insert>
If we want to add after the label content:
<map:insert type="label" location="content">
<map:transformer type="gather-links"/>
</map:insert>
What here is missing is the declaration of the inserts.
Like when calling transformers I have to define the Transformer type,
here I have to define the Inserter type.
<map:components>
<map:inserters>
<map:inserter name="label"
class="org.apache.cocoon.inserters.LabelInserter"/>
<map:inserter name="index"
class="org.apache.cocoon.inserters.IndexInserter"/>
</map:inserters>
</map:components>
Where an Inserter is a Components that gets a Pipeline and a series of
Pipeline Components and inserts those in the pipeline as loaction indicates.
public void insert(Pipeline p,
PipelineComponent[] components,
String location){
...
}
The sitemap would call all inserters on all pipelines prior to starting.
This though needs additions to our contracts. What about making a
special pipeline that can be aspected?
<map:pipes default="aspected">
<map:pipe name="aspected"
src="org.apache.cocoon.components.pipeline.impl.AspectedProcessingPipeline">
<param name="label:content" value="call-resource:link-gatherer"/>
<param name="index:first" value="call-resource:link-gatherer"/>
</map:pipe>
</map:pipes>
Bleah :-P
Let's start with the automatic-or-Transformer thing for now, then
eventually evolve out of it.
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: Aspect-based pipelines and link view ( Re: Link view goodness)
Posted by Upayavira <uv...@upaya.co.uk>.
Guys,
> The link stuff is a cross-cutting concern. This thread has IMHO shown
> how aspects can be easily added to the sitemap, and effectively used.
> Let's see...
> >>>- We're abusing the name 'transformer', since nothing is
> >>>transformed.
> >>> If we're really going to go this way, let's define a new sitemap
> >>> element, <map:link-gatherer/>.
> >>
> >>There are transformers that do not transform, it's not unusual,
> >
> > I can't think of any others?
>
> some OTOMH, maybe not 100% correct:
>
> LogTransformer
> XMLFormTransformer
> WriteDOMSessionTransformer
> SourceWritingTransformer
Yup. Now, I've got an (untested) LinkGatheringTransformer ready and compiling. It
shouldn't be much work to test it and get it going.
> > <snip links to stuff I mostly missed - thanks>
> >
> >>"
> >>So basically we are adding a contract to the sitemap, by saying that
> >>each sitemap implementation has to provide a list of links if
> >>requested to (as seen above). "
> >>
> >>As you state, a Transformer does not feel right. In fact, a sitemap
> >>has now a new contract that it has to give links. The question is:
> >>how can it be made more versatile? Who can we tell the pipeline
> >>where we want the link gathering to occur?
> >>
> >>What about a named pipeline that is inserted by the link gatherer
> >>where it gets the links? What about using a spacial label to
> >>indicate where to gather links?
What is a named pipeline? How would the link gatherer (or rather the bean) 'insert a
named pipeline'?
> > Hmm.. interesting. Perhaps we just need to augment Resources a bit:
> >
> > <map:resource name="gather-links" from-position="content">
> > <!-- Any required link munging -->
> > <map:transformer type="gather-links"/>
> > </map:resource>
>
> Cool, you have put my words in code, adding that last bit that makes
> them worthwile :-) This really looks like some sort of final solution,
> intriguing.
So how does this work? I don't get it. You're specifying a resource, but presumably
you're going to still have to insert it somehow?
> Hmmm, it also has to do with the "named pipelines" thread, or the
> pipeline==reusable_component one that Stefano had started.
>
> Seems like simply adding this capability to resources is nice. We
> could similarly make a link-view that uses the same transformer and a
> serializer. In this way it could also be compatible with the 3pass
> method. Hmmm...
>
> > Ie, a Resource inserted in each pipeline after the 'content' label.
> > Rather AOP'ish.
>
> Yup. As link gathering is a cross-cutting concern, it also makes sense
> conceptually.
So you're saying, with a resource that has a 'from-position' attribute, that specifies
after which label it should be inserted? That makes sense. So you only have to have
the resource once per sitemap, rather than having to insert it into every pipeline.
But - what if the pipeline itself needs modifying to expose links from within PDFs for
example. The LinkGatheringTransformer I have coded has two modes, one where it
just hunts for href, src and xlink attributes, and the other that searches for attributes
in the http://apache.org/cocoon/link-gatherer/1.0 namespace (probably to be used
with a 'link' prefix). This latter kind is required for gathering links that don't conform to
the href, src or xlink conventions. Just auto-inserting a link gatherer wouldn't work in
this case.
> The thing here is how it's called by the sitemap engine. There is no
> explicit call in the pipelines, but instead a "from-position"
> attribute. It could easily have also a serializer, and in this way it
> would terminate all pipelines... but it's like fa view in this case...
>
> But a view then would become an AOPish resource with a serializer
> called only on certain conditions. Let's call them aspects instead of
> resources.
>
> So:
> - <pipelines> are called per request one after the other till the
> sitemap exits - <resources> are sitemap snippets called by the
> pipelines - <views> are exit points that get called at a particular
> label
> (effectively a hard-wired AOP feature) by the sitemap
>
> Pipelines and resources are effectively the same thing not that there
> is the cocoon protocol.
>
> What remains is the <views> part, that has introduce pipeline-stage
> metadata, as a label. It's an aspect that gets called when that
> particular condition is met (I won't use AOP terminology that I
> personally don't yet like)
>
> So we can generalize it, and add configurability to the view mechanism
> to specify other conditions.
>
> <map:view name="content" from-label="content">
> <map:serialize type="xml"/>
> </map:view>
>
> becomes:
>
> <map:view name="content" type="from-label"
> test="content">
> <map:serialize type="xml"/>
> </map:view>
>
> This makes it possible to make a different position where to start
> from...
>
> What can also be made configurable is *when*, in which condition, it's
> triggered, but the logic has to be inverted.
>
> Now we say: "when the view is triggered, start at a label"
> After it could be: "when the view is triggered, start at position"
> Instead we need: "when the position is met, check if it has to be
> triggered".
>
> Here is an example that uses this "inverted" AOPish system for views.
>
> The following adds two aspects:
> - an aspect gets called from every content position and gathers
> links. - the other one gets called from every content position. If
> the
> request has a cocoon-view=links, then the links are serialized.
>
> <map:aspects>
> <map:aspect type="from-label" test="content">
> <!-- Any required link munging -->
> <map:transformer type="gather-links"/>
> </map:aspect>
> <map:aspect type="from-label" test="content">
> <map:action type="request-param">
> <map:param name="cocoon-view" value="link">
> <map:serializer type="links"/>
> </map:action>
> </map:aspect>
> </map:aspects>
>
> This would make it very easy to add security-based checks, logging, or
> any other stuff.
>
> <map:aspects>
> <map:aspect type="pipeline" test="start">
> <map:action type="check-security"/>
> </map:aspect>
> <map:aspect type="pipeline" test="all">
> <map:transformer type="logger"/>
> </map:aspect>
> <map:aspect type="error" test="all">
> <map:action type="notify-admin"/>
> </map:aspect>
> </map:aspects>
>
> What do others think?
>
> Is it already possible to do this today with other components and
> skillful pipeline writing? For doing it at the beginning or at the end
> of a request it's possible to have an entry-point pieline that has pre
> and post processing, but to add stuff *inside* other pipelines? I
> think it cannot be done today.
I'm afraid you left me completely behind there. I've not really yet understood what
AOP is, and your ideas go far further than my Cocoon implementation skills currently
allow.
I'd quite like to find something that can be implemented reasonably short term, and
then explore these more far-reaching ideas as time passes (and as the size and
capacity of my brain increases).
Are you guys interested for the time being in a LinkGatheringTransformer as
described above? Or is there something not too far away that we can do now to
gather links?
Regards, Upayavira
Aspect-based pipelines and link view ( Re: Link view goodness (Re:
residuals of MIME type bug ?) )
Posted by Nicola Ken Barozzi <ni...@apache.org>.
The link stuff is a cross-cutting concern. This thread has IMHO shown
how aspects can be easily added to the sitemap, and effectively used.
Let's see...
Jeff Turner wrote, On 29/06/2003 14.48:
> On Sun, Jun 29, 2003 at 11:34:01AM +0200, Nicola Ken Barozzi wrote:
>
>>Jeff Turner wrote, On 29/06/2003 8.03:
>
> ....
>>>- We're abusing the name 'transformer', since nothing is transformed.
>>> If we're really going to go this way, let's define a new sitemap
>>> element, <map:link-gatherer/>.
>>
>>There are transformers that do not transform, it's not unusual,
>
> I can't think of any others?
some OTOMH, maybe not 100% correct:
LogTransformer
XMLFormTransformer
WriteDOMSessionTransformer
SourceWritingTransformer
> <snip links to stuff I mostly missed - thanks>
>
>>"
>>So basically we are adding a contract to the sitemap, by saying that
>>each sitemap implementation has to provide a list of links if requested
>>to (as seen above).
>>"
>>
>>As you state, a Transformer does not feel right. In fact, a sitemap has
>>now a new contract that it has to give links. The question is: how can
>>it be made more versatile? Who can we tell the pipeline where we want
>>the link gathering to occur?
>>
>>What about a named pipeline that is inserted by the link gatherer where
>>it gets the links? What about using a spacial label to indicate where to
>>gather links?
>
> Hmm.. interesting. Perhaps we just need to augment Resources a bit:
>
> <map:resource name="gather-links" from-position="content">
> <!-- Any required link munging -->
> <map:transformer type="gather-links"/>
> </map:resource>
Cool, you have put my words in code, adding that last bit that makes
them worthwile :-)
This really looks like some sort of final solution, intriguing.
Hmmm, it also has to do with the "named pipelines" thread, or the
pipeline==reusable_component one that Stefano had started.
Seems like simply adding this capability to resources is nice. We could
similarly make a link-view that uses the same transformer and a
serializer. In this way it could also be compatible with the 3pass
method. Hmmm...
> Ie, a Resource inserted in each pipeline after the 'content' label.
> Rather AOP'ish.
Yup. As link gathering is a cross-cutting concern, it also makes sense
conceptually.
The thing here is how it's called by the sitemap engine. There is no
explicit call in the pipelines, but instead a "from-position" attribute.
It could easily have also a serializer, and in this way it would
terminate all pipelines... but it's like a view in this case...
But a view then would become an AOPish resource with a serializer called
only on certain conditions. Let's call them aspects instead of resources.
So:
- <pipelines> are called per request one after the other till the
sitemap exits
- <resources> are sitemap snippets called by the pipelines
- <views> are exit points that get called at a particular label
(effectively a hard-wired AOP feature) by the sitemap
Pipelines and resources are effectively the same thing not that there is
the cocoon protocol.
What remains is the <views> part, that has introduce pipeline-stage
metadata, as a label. It's an aspect that gets called when that
particular condition is met (I won't use AOP terminology that I
personally don't yet like)
So we can generalize it, and add configurability to the view mechanism
to specify other conditions.
<map:view name="content" from-label="content">
<map:serialize type="xml"/>
</map:view>
becomes:
<map:view name="content" type="from-label"
test="content">
<map:serialize type="xml"/>
</map:view>
This makes it possible to make a different position where to start from...
What can also be made configurable is *when*, in which condition, it's
triggered, but the logic has to be inverted.
Now we say: "when the view is triggered, start at a label"
After it could be: "when the view is triggered, start at position"
Instead we need: "when the position is met, check if it has to be
triggered".
Here is an example that uses this "inverted" AOPish system for views.
The following adds two aspects:
- an aspect gets called from every content position and gathers links.
- the other one gets called from every content position. If the
request has a cocoon-view=links, then the links are serialized.
<map:aspects>
<map:aspect type="from-label" test="content">
<!-- Any required link munging -->
<map:transformer type="gather-links"/>
</map:aspect>
<map:aspect type="from-label" test="content">
<map:action type="request-param">
<map:param name="cocoon-view" value="link">
<map:serializer type="links"/>
</map:action>
</map:aspect>
</map:aspects>
This would make it very easy to add security-based checks, logging, or
any other stuff.
<map:aspects>
<map:aspect type="pipeline" test="start">
<map:action type="check-security"/>
</map:aspect>
<map:aspect type="pipeline" test="all">
<map:transformer type="logger"/>
</map:aspect>
<map:aspect type="error" test="all">
<map:action type="notify-admin"/>
</map:aspect>
</map:aspects>
What do others think?
Is it already possible to do this today with other components and
skillful pipeline writing? For doing it at the beginning or at the end
of a request it's possible to have an entry-point pieline that has pre
and post processing, but to add stuff *inside* other pipelines? I think
it cannot be done today.
>>Just food for thought.
>
> Tasty..
I really appreciate it, and how you manage to work with me even when I'm
a bit touchy. Thanks :-)
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Sun, Jun 29, 2003 at 11:34:01AM +0200, Nicola Ken Barozzi wrote:
> Jeff Turner wrote, On 29/06/2003 8.03:
...
> >- We're abusing the name 'transformer', since nothing is transformed.
> > If we're really going to go this way, let's define a new sitemap
> > element, <map:link-gatherer/>.
>
> There are transformers that do not transform, it's not unusual,
I can't think of any others?
<snip links to stuff I mostly missed - thanks>
> "
> So basically we are adding a contract to the sitemap, by saying that
> each sitemap implementation has to provide a list of links if requested
> to (as seen above).
> "
>
> As you state, a Transformer does not feel right. In fact, a sitemap has
> now a new contract that it has to give links. The question is: how can
> it be made more versatile? Who can we tell the pipeline where we want
> the link gathering to occur?
>
> What about a named pipeline that is inserted by the link gatherer where
> it gets the links? What about using a spacial label to indicate where to
> gather links?
Hmm.. interesting. Perhaps we just need to augment Resources a bit:
<map:resource name="gather-links" from-position="content">
<!-- Any required link munging -->
<map:transformer type="gather-links"/>
</map:resource>
Ie, a Resource inserted in each pipeline after the 'content' label.
Rather AOP'ish.
> Just food for thought.
Tasty..
--Jeff
> --
> Nicola Ken Barozzi nicolaken@apache.org
> - verba volant, scripta manent -
> (discussions get forgotten, just code remains)
> ---------------------------------------------------------------------
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Jeff Turner wrote, On 29/06/2003 8.03:
...
> I still have the feeling that a link-gatherer transformer is mixing
> concerns a bit, and that two-pass is conceptually nicer:
>
> - We're abusing the name 'transformer', since nothing is transformed.
> If we're really going to go this way, let's define a new sitemap
> element, <map:link-gatherer/>.
There are transformers that do not transform, it's not unusual,
although, since the sitemap has a new contract on links (see at the
bottom), it might make sense.
> - Link gathering is irrelevant for online situations, so we pay some
> performance penalty having a link-gatherer transformer. This
> illustrates why I think it mixes concerns.
Exactly.
> - It's easy to forget to define a link-gatherer transformer for new
> pipelines. Link-view is cross-cutting and doesn't have this
> problem.
Again, exactly.
> I'm not very familiar with the code; is there some cost in keeping the
> two-pass CLI alive, in the faint hope that caching comes to its rescue
> one day?
Actually it was three-pass.
http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=104013686220328&w=2
>>Thanks for engaging with me on this - I appreciate it.
>
>
> Thank _you_; an improved CLI will make Forrest significantly more
> usable.
For your pleasure, and of interested parties, the previous threads:
http://marc.theaimsgroup.com/?t=102725710300001&r=1&w=2
http://marc.theaimsgroup.com/?t=104013701500006&r=1&w=2
http://marc.theaimsgroup.com/?t=104609314900002&r=1&w=2
http://marc.theaimsgroup.com/?t=104887033400005&r=1&w=2
And a couple of mails:
http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=104610949203967&w=2
http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=104679840022563&w=2
http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=104687731531754&w=2
The last mail in particular explains the current new-CLI method:
"
So basically we are adding a contract to the sitemap, by saying that
each sitemap implementation has to provide a list of links if requested
to (as seen above).
"
As you state, a Transformer does not feel right. In fact, a sitemap has
now a new contract that it has to give links. The question is: how can
it be made more versatile? Who can we tell the pipeline where we want
the link gathering to occur?
What about a named pipeline that is inserted by the link gatherer where
it gets the links? What about using a spacial label to indicate where to
gather links?
Just food for thought.
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Vadim Gritsenko <va...@verizon.net>.
Jeff Turner wrote:
>I'm not very familiar with the code; is there some cost in keeping the
>two-pass CLI alive, in the faint hope that caching comes to its rescue
>one day?
>
Guys,
Before you implement some approach here... Let me suggest something.
Right now sitemap implementation automatically adds link gatherer to the
pipeline when it is invoked by CLI. This link gatherer is in fact is
"hard-coded links view". I suggest to replace this "hard-coded links
view" a.k.a link gatherer with the "real" links view, BUT attach it as a
tee to a main pipeline instead of running it as a pipeline by itself. As
a result, links view "baby" will be used, two-pass "water" will be
drained, and sitemap syntax will stay the same. Moreover, the links view
will be still accessible from the outside, meaning that you can spider
the site using out-of-the-process spiders.
Example:
Given the pipeline:
G --> T1 (label="content") --> T2 --> S,
And the links view:
from-label="content" --> T3 --> LinkSerializer,
The pipeline built for the CLI request should be:
G --> T1 --> Tee --> T2 --> S --> OutputStream
\
--> LinkSerializer --> NullOutputStream
\
--> List of links in environment
In one request, you will get:
* Regular output of the pipeline which will go to the destination Source
* List of links in the environment which is what link gatherer was made for
Comments?
Vadim
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Sat, Jun 28, 2003 at 03:38:55PM +0100, Upayavira wrote:
...
> Okay. How about defining a namespace <links:link href="xxxx"/> which
> gets consumed by the transformer, that way you choose in your
> previous XSLT which links you want to be spidered by presenting the
> links in that <links> namespace (and then repeat them for the sake
> of the output).
Sounds good. So you mean, eg, transforming <a href="..."> into <a
href="..." link:href="...">, and the gather-links transformer uses
the link:href attribute?
...
> Now the only question that remains is whether to have an implicit
> gatherer if no explicit one is specified. I'd probably say no, as
> other discussions have erred away from hidden things like that.
+1
> I think that telling the sitemap where your links are is a pretty
> reasonable adjustment to your site. In fact, we could have two
> transformers - one that just looks for hrefs and xlinks, and another
> that uses a links namespace - the former would make it real easy to
> convert your site for spidering, and the latter providing a method
> to do complex link management.
+1, was just going to suggest that.
> Another question - do we still leave link view (two pass) link
> following in the CLI? Or does this method deprecate and thus replace
> it?
I still have the feeling that a link-gatherer transformer is mixing
concerns a bit, and that two-pass is conceptually nicer:
- We're abusing the name 'transformer', since nothing is transformed.
If we're really going to go this way, let's define a new sitemap
element, <map:link-gatherer/>.
- Link gathering is irrelevant for online situations, so we pay some
performance penalty having a link-gatherer transformer. This
illustrates why I think it mixes concerns.
- It's easy to forget to define a link-gatherer transformer for new
pipelines. Link-view is cross-cutting and doesn't have this
problem.
I'm not very familiar with the code; is there some cost in keeping the
two-pass CLI alive, in the faint hope that caching comes to its rescue
one day?
> Thanks for engaging with me on this - I appreciate it.
Thank _you_; an improved CLI will make Forrest significantly more
usable.
--Jeff
> Regards, Upayavira
>
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Upayavira <uv...@upaya.co.uk>.
Jeff wrote:
> > So are you saying you can manage without the XSLT stage?
>
> I'm not sure, perhaps you can advise. In Forrest we filter the links
> to:
>
> - Remove API doc links
> - Remove links to directories, which break the CLI
> - Remove image links that have been hacked to work with FOP
>
> 1) belongs in cli.xconf. Perhaps the new CLI handles 2) better than
> the original. I think 3) is obsolete, as LinkSerializer ignores
> XSL:FO-namespaced links anyway.
>
> > Perhaps I should explain what I had in mind a bit more with that - I
> > guess I would call it a tee, a pipeline element with one input and
> > two outputs. The input is passed unchanged on through to the next
> > stage in the pipeline. But it is also passed through an XSLT before
> > links are gathered from it.
>
> I'd call it a hack ;) Why favour XSLT and not STX, or any other
> transformer? What about XSLT parameters? etc. If people need XSLT,
> let them use a link view. I'd suggest just sticking with the basics:
> <map:transform type="gather-links"/>
Okay. How about defining a namespace <links:link href="xxxx"/> which gets
consumed by the transformer, that way you choose in your previous XSLT which links
you want to be spidered by presenting the links in that <links> namespace (and then
repeat them for the sake of the output).
This would be an extremely simple transformer to write. Beyond writing the
transformer, it would take a minimal amount (1/2 hour) of changes to the rest of the
CLI.
> Which isn't a hack. In fact it would be great for Forrest, because we
> only have a few matchers where links are relevant. All the cocoon:
> and image pipelines could go without.
Yup.
> Also, it resolves another little dilemma I've had with link views.
> It's all very well having the notion of a cross-cutting 'view', but
> there's no way to override the 'view' for a specific pipeline. With
> an explicit gather-links transformer, one could have different link
> analysis for each pipeline. A *.css pipeline could list @import's as
> links, for example.
Great.
> > > It certainly fixes the hard-wired'ness problem you mention above
> > > (that 'content' != XML before the serializer).
> >
> > And it sounds as if it could be a trivial solution.
>
> 'Solves' the cocoon: sub-pipeline problem too.
Yup.
Now the only question that remains is whether to have an implicit gatherer if no
explicit one is specified. I'd probably say no, as other discussions have erred away
from hidden things like that.
I think that telling the sitemap where your links are is a pretty reasonable adjustment
to your site. In fact, we could have two transformers - one that just looks for hrefs and
xlinks, and another that uses a links namespace - the former would make it real easy
to convert your site for spidering, and the latter providing a method to do complex link
management.
Another question - do we still leave link view (two pass) link following in the CLI? Or
does this method deprecate and thus replace it?
Thanks for engaging with me on this - I appreciate it.
Regards, Upayavira
Type-aware Views (Re: Link view goodness)
Posted by Jeff Turner <je...@apache.org>.
On Sun, Jun 29, 2003 at 09:08:14PM +1200, Conal Tuohy wrote:
> Jeff Turner wrote:
>
> > > That's an issue I've come up against too - it seems that views are
> > > still too "tangled" up with labels and can't cut across pipelines
> > > properly. At least, that's how I understand it - maybe I'm missing
> > > something?
> >
> > I think labels and Views are independent of each other. You can have
> > a view defined with 'from-position', and not use labels. Labels are
> > just generic markers, with nothing to say they're only useful for
> > defining views.
>
> But with from-position you can have only "first" and "last" which is
> even more restrictive than labels. If you want to do anything very
> sophisticated don't you need labels?
Yes, labels and positions. What else could there be?
> > Views give _every_ public URL in a sitemap an alternative form. If
> > you only need an alternative form of some URLs, then that can be done
> > just as you've described above, with a request-param selector.
>
> So ... I could just have use a RequestParamSelector to create my
> different views for the crawler? Damn!
I doubt it. I was just describing when you'd want to use views at all.
The old CLI chose to use views, which means there's no option for
per-pipeline customization.
> My problem was that I wanted to use Lucene to index a "content" view of
> 2 different pipelines, one of them based on TEI and another on HTML. In
> the case of the TEI pipeline I didn't want to convert the TEI to HTML
> first and then produce a "content" view based on an HTML-ized view of
> the TEI - I wanted an indexable view of the TEI. This is the same issue
> as you mention below:
>
> > The problem is that Views don't know the type of data they're
> > getting. If we have a view with from-label="content", we know it's
> > content, but what _type_ of content? What schema? What
> > transformation can we apply to create a links-view of this content?
>
> If you could create more than one view with the same name, then we
> could use labels to specify the schema:
>
> e.g. 2 pipelines containing:
> ...
> <map:generate src="{1}.xml" label="tei"/>
> ...
>
> and
>
> <map:transform src="blah-to-html.xsl" label="html"/>
>
> ... and 2 views called "content", one with from-label="tei" and the
> other with from-label="html".
Technically that's more or less the solution. I think a cleaner way
of presenting it is to have one view that interprets different kinds
of data differently:
<map:view name="links" from-position="content">
<map:select type="xml-type">
<map:when test="html">
<map:transform src="html2whatever.xsl"/>
</map:when>
<map:when test="tei">
<map:transform src="tei2whatever.xsl"/>
</map:when>
</map:select>
</map:view>
So, treating 'type' as a property of a sitemap component, independent
of labels. The xml-type selector would somehow discover the type of
XML emitted by its upstream component.
--Jeff
> Cheers
>
> Con
>
RE: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Conal Tuohy <co...@paradise.net.nz>.
Jeff Turner wrote:
> > That's an issue I've come up against too - it seems that views are
> > still too "tangled" up with labels and can't cut across pipelines
> > properly. At least, that's how I understand it - maybe I'm missing
> > something?
>
> I think labels and Views are independent of each other. You
> can have a
> view defined with 'from-position', and not use labels.
> Labels are just
> generic markers, with nothing to say they're only useful for defining
> views.
But with from-position you can have only "first" and "last" which is even
more restrictive than labels. If you want to do anything very sophisticated
don't you need labels?
> Views give _every_ public URL in a sitemap an alternative
> form. If you
> only need an alternative form of some URLs, then that can be
> done just as
> you've described above, with a request-param selector.
So ... I could just have use a RequestParamSelector to create my different
views for the crawler? Damn!
My problem was that I wanted to use Lucene to index a "content" view of 2
different pipelines, one of them based on TEI and another on HTML. In the
case of the TEI pipeline I didn't want to convert the TEI to HTML first and
then produce a "content" view based on an HTML-ized view of the TEI - I
wanted an indexable view of the TEI. This is the same issue as you mention
below:
> The problem is that Views don't know the type of data they're getting.
> If we have a view with from-label="content", we know it's content, but
> what _type_ of content? What schema? What transformation
> can we apply
> to create a links-view of this content?
If you could create more than one view with the same name, then we could use
labels to specify the schema:
e.g. 2 pipelines containing:
...
<map:generate src="{1}.xml" label="tei"/>
...
and
<map:transform src="blah-to-html.xsl" label="html"/>
... and 2 views called "content", one with from-label="tei" and the other
with from-label="html".
Cheers
Con
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Sun, Jun 29, 2003 at 05:36:45PM +1200, Conal Tuohy wrote:
> Jeff Turner wrote:
>
> <snip/>
>
> > Also, it resolves another little dilemma I've had with link views.
> > It's all very well having the notion of a cross-cutting 'view', but
> > there's no way to override the 'view' for a specific pipeline. With
> > an explicit gather-links transformer, one could have different link
> > analysis for each pipeline. A *.css pipeline could list @import's as
> > links, for example.
>
> That's an issue I've come up against too - it seems that views are
> still too "tangled" up with labels and can't cut across pipelines
> properly. At least, that's how I understand it - maybe I'm missing
> something?
I think labels and Views are independent of each other. You can have a
view defined with 'from-position', and not use labels. Labels are just
generic markers, with nothing to say they're only useful for defining
views.
> For instance I couldn't see how to have 2 pipelines share a view (i.e.
> both support a view) unless the 2 pipelines had a common stage
> somewhere.
>
> I've always wondered why views weren't implemented using a Selector?
>
> <map:select type="view">
> <map:when test="links">
> <map:transform src="convert-format-X-to-links.xsl"/>
> <map:serialize type="links"/>
> </map:when>
> <map:when test="content">
> <map:transform src="convert-format-X-to-HTML-xsl"/>
> <map:serialize type="html"/>
> </map:when>
> </map:select>
Views give _every_ public URL in a sitemap an alternative form. If you
only need an alternative form of some URLs, then that can be done just as
you've described above, with a request-param selector.
The problem is that Views don't know the type of data they're getting.
If we have a view with from-label="content", we know it's content, but
what _type_ of content? What schema? What transformation can we apply
to create a links-view of this content?
That's why I'm looking forward to Cocoon 4.0, which will have strongly
typed pipelines. Then the links view can see what kind of content its
getting (say *.css), and apply an appropriate transformation to extract
links (@import'ed files). Given the current release rate, Cocoon 4.0 is
due in early 2030.
--Jeff
> In this way different pipelines could have quite different views,
> without sharing a commonly-labelled component. I guess this is more
> verbose than the current approach, where the view transformations are
> attached by name using a label, but for some reason the label approach
> reminds me powerfully of GOTO.
>
> Cheers
>
> Con
>
RE: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Conal Tuohy <co...@paradise.net.nz>.
Jeff Turner wrote:
<snip/>
> Also, it resolves another little dilemma I've had with link
> views. It's
> all very well having the notion of a cross-cutting 'view',
> but there's no
> way to override the 'view' for a specific pipeline. With an explicit
> gather-links transformer, one could have different link
> analysis for each
> pipeline. A *.css pipeline could list @import's as links,
> for example.
That's an issue I've come up against too - it seems that views are still too
"tangled" up with labels and can't cut across pipelines properly. At least,
that's how I understand it - maybe I'm missing something?
For instance I couldn't see how to have 2 pipelines share a view (i.e. both
support a view) unless the 2 pipelines had a common stage somewhere.
I've always wondered why views weren't implemented using a Selector?
<map:select type="view">
<map:when test="links">
<map:transform src="convert-format-X-to-links.xsl"/>
<map:serialize type="links"/>
</map:when>
<map:when test="content">
<map:transform src="convert-format-X-to-HTML-xsl"/>
<map:serialize type="html"/>
</map:when>
</map:select>
In this way different pipelines could have quite different views, without
sharing a commonly-labelled component. I guess this is more verbose than the
current approach, where the view transformations are attached by name using
a label, but for some reason the label approach reminds me powerfully of
GOTO.
Cheers
Con
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Sat, Jun 28, 2003 at 11:07:45AM +0100, Upayavira wrote:
> On 28 Jun 2003 at 18:45, Jeff Turner wrote:
...
> > > <map:match pattern="page.html">
> > > <map:generate src="page.xml"/>
> > > <map:transform src="before-links.xsl"/>
> > > <map:transform type="gather-links" src="identify-links.xsl"/>
> > > <map:transform src="after-links.xsl"/> <map:serialize/>
> > > </map:match>
> > >
> > > So there's no hidden link gatherer. And you've got a single xslt to
> > > filter, etc. Not specifying src="xxx" skips the xsl stage. The
> > > output of this xsl would be xml conforming to a predefined
> > > namespace.
> >
> > Having eliminated the dont-follow-these-links use-case, I don't see a
> > use-case for XSLT transformations, so it simplifies to
> >
> > <map:transform type="gather-links"/>
>
> So are you saying you can manage without the XSLT stage?
I'm not sure, perhaps you can advise. In Forrest we filter the links to:
- Remove API doc links
- Remove links to directories, which break the CLI
- Remove image links that have been hacked to work with FOP
1) belongs in cli.xconf. Perhaps the new CLI handles 2) better than the
original. I think 3) is obsolete, as LinkSerializer ignores
XSL:FO-namespaced links anyway.
> Perhaps I should explain what I had in mind a bit more with that - I
> guess I would call it a tee, a pipeline element with one input and two
> outputs. The input is passed unchanged on through to the next stage in
> the pipeline. But it is also passed through an XSLT before links are
> gathered from it.
I'd call it a hack ;) Why favour XSLT and not STX, or any other
transformer? What about XSLT parameters? etc. If people need XSLT, let
them use a link view. I'd suggest just sticking with the basics:
<map:transform type="gather-links"/>
Which isn't a hack. In fact it would be great for Forrest, because we
only have a few matchers where links are relevant. All the cocoon: and
image pipelines could go without.
Also, it resolves another little dilemma I've had with link views. It's
all very well having the notion of a cross-cutting 'view', but there's no
way to override the 'view' for a specific pipeline. With an explicit
gather-links transformer, one could have different link analysis for each
pipeline. A *.css pipeline could list @import's as links, for example.
> > It certainly fixes the hard-wired'ness problem you mention above (that
> > 'content' != XML before the serializer).
>
> And it sounds as if it could be a trivial solution.
'Solves' the cocoon: sub-pipeline problem too.
--Jeff
>
> Upayavira
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Upayavira <uv...@upaya.co.uk>.
On 28 Jun 2003 at 18:45, Jeff Turner wrote:
> On Sat, Jun 28, 2003 at 07:29:49AM +0100, Upayavira wrote:
> > On 28 Jun 2003 at 11:59, Jeff Turner wrote:
> ...
> > Okay. For the CLI, the cli.xconf file is the equivalent of the
> > web.xml and the user agent.
> >
> > Now, normally the user agent requests a URI, and that's it. It is up
> > to the user agent as to what to do with that URI.
>
> Oh I see. Yep, makes sense that the 'user agent' be the one who
> decides whether or not to chase down links.
>
> > Are you saying that you want to put the configuration as to where
> > pages should be placed into the sitemap?
>
> No, that's the user agent's (CLI's) business.
Good.
> > Or an alternative would be to ask: can you always do your link view
> > with a single XSLT stage? If so:
> >
> > <map:match pattern="page.html">
> > <map:generate src="page.xml"/>
> > <map:transform src="before-links.xsl"/>
> > <map:transform type="gather-links" src="identify-links.xsl"/>
> > <map:transform src="after-links.xsl"/> <map:serialize/>
> > </map:match>
> >
> > So there's no hidden link gatherer. And you've got a single xslt to
> > filter, etc. Not specifying src="xxx" skips the xsl stage. The
> > output of this xsl would be xml conforming to a predefined
> > namespace.
>
> Having eliminated the dont-follow-these-links use-case, I don't see a
> use-case for XSLT transformations, so it simplifies to
>
> <map:transform type="gather-links"/>
So are you saying you can manage without the XSLT stage? Perhaps I should
explain what I had in mind a bit more with that - I guess I would call it a tee, a pipeline
element with one input and two outputs. The input is passed unchanged on through
to the next stage in the pipeline. But it is also passed through an XSLT before links
are gathered from it.
Are you saying you can manage without this?
> It certainly fixes the hard-wired'ness problem you mention above (that
> 'content' != XML before the serializer).
And it sounds as if it could be a trivial solution.
Upayavira
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Sat, Jun 28, 2003 at 07:29:49AM +0100, Upayavira wrote:
> On 28 Jun 2003 at 11:59, Jeff Turner wrote:
...
> Okay. For the CLI, the cli.xconf file is the equivalent of the web.xml and the user
> agent.
>
> Now, normally the user agent requests a URI, and that's it. It is up to the user agent
> as to what to do with that URI.
Oh I see. Yep, makes sense that the 'user agent' be the one who decides
whether or not to chase down links.
> Are you saying that you want to put the configuration as to where pages
> should be placed into the sitemap?
No, that's the user agent's (CLI's) business.
...
> Yup. The primary aim was to reduce the number of page generations. And there was
> an element of hack here - particularly in the 'hard-wired'ness of the LinkGatherer.
...
> Or an alternative would be to ask: can you always do your link view
> with a single XSLT stage? If so:
>
> <map:match pattern="page.html">
> <map:generate src="page.xml"/>
> <map:transform src="before-links.xsl"/>
> <map:transform type="gather-links" src="identify-links.xsl"/>
> <map:transform src="after-links.xsl"/>
> <map:serialize/>
> </map:match>
>
> So there's no hidden link gatherer. And you've got a single xslt to filter, etc. Not
> specifying src="xxx" skips the xsl stage. The output of this xsl would be xml
> conforming to a predefined namespace.
Having eliminated the dont-follow-these-links use-case, I don't see a
use-case for XSLT transformations, so it simplifies to
<map:transform type="gather-links"/>
It certainly fixes the hard-wired'ness problem you mention above (that
'content' != XML before the serializer).
--Jeff
>
> Regards, Upayavira
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Upayavira <uv...@upaya.co.uk>.
On 28 Jun 2003 at 11:59, Jeff Turner wrote:
> Conceptually, I link the link-view because:
>
> 1) Links are URIs
> 2) The sitemap is 100% in control of the URI space
>
> implying:
>
> 3) The sitemap ought to be in control of link URI manipulation, not
> some external cli.xconf file.
Okay. For the CLI, the cli.xconf file is the equivalent of the web.xml and the user
agent.
Now, normally the user agent requests a URI, and that's it. It is up to the user agent
as to what to do with that URI. Are you saying that you want to put the configuration
as to where pages should be placed into the sitemap? And which URIs should be
rendered? If so, how would you do this?
Thing is, for me, that means hardwiring the URIs you want to render into your site,
and doesn't allow for a dynamic regeneration of different parts of the site.
> Now for practicalities:
>
> I like the fact that the sitemap writer has full control over what is
> considered a link, and what those links look like. An invisible
> linkgatherer transformer effectively hardcodes:
>
> <map:serializer name="links"
> src="org.apache.cocoon.serialization.LinkSerializer">
> <encoding>ISO-8859-1</encoding>
> </map:serializer>
> <map:view name="links" from-position="last">
> <map:serialize type="links"/>
> </map:view>
Yup. The primary aim was to reduce the number of page generations. And there was
an element of hack here - particularly in the 'hard-wired'ness of the LinkGatherer.
It has to be said that the link gatherer uses the same approach as the LinkTranslator,
which is used by the 'mime-type checking' code. That's where I got the idea.
> There are various points of flexibility that the links view allows:
>
> Alternative Link schemes
> ------------------------
>
> If the user's XML doesn't happen to use XLink or @href for linking,
> they would implement an alternative to LinkSerializer.
>
> For example, imagine we want to render only PDFs. The last XSLT in
> our pipeline would produce xsl:fo. The standard LinkSerializer
> doesn't know about fo:external-link elements. Even if it did, we'd
> want to filter out links to images, since PDFs have images inlined.
> What is an image? That's up to the sitemap writer.
>
> Encoding
> --------
> When serializing links in Japanese or something, wouldn't tweaking the
> <encoding> tag be necessary?
>
> Filtering unwanted links
> ------------------------
> We can filter out unwanted links, with arbitrary precision (eg using
> XPath expressions to determine what to throw out). In Forrest we use
> <xsl:when test="contains(., 'api/')"> to filter out javadoc links.
> Eventually, 'api/' will be determined at runtime, by querying an input
> module that reads a forrest.xml config file.
I can (and already could) see these benefits. I would like to see a way to meet both of
our requirements (a link view and single pass generation). Now, caching might be the
simplest way. Or an alternative would be to ask: can you always do your link view with
a single XSLT stage? If so:
<map:match pattern="page.html">
<map:generate src="page.xml"/>
<map:transform src="before-links.xsl"/>
<map:transform type="gather-links" src="identify-links.xsl"/>
<map:transform src="after-links.xsl"/>
<map:serialize/>
</map:match>
So there's no hidden link gatherer. And you've got a single xslt to filter, etc. Not
specifying src="xxx" skips the xsl stage. The output of this xsl would be xml
conforming to a predefined namespace.
> I hope I've convinced you :) Certainly for simpler needs, hardcoding
> a LinkGathererTransformer is fine, but in general (and I hope where
> Forrest is going) we need the full power of a link view.
I've always been convinced - just don't like the double pass.
Regards, Upayavira
I think there's a place for both, but I'd like to get it
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Upayavira wrote, On 28/06/2003 17.00:
> Nicola Ken,
>
>>Sorry Jeff, but I don't have time nor energy to dwelve into this
>>discussion further. I'm getting a bit tired about it too. Don't get me
>>wrong, it's not about you, it's just that sometimes one loses interest
>>in some things.
>
> FWIW, I'm pleased that Jeff is prepared to go along with these discussions - I think
> our original discussions only went so far. We got it down to one pass, and were pretty
> happy - we didn't really engage further with what the real consequences of that were,
> and what we potentially lost. And I think because we didn't do this, we haven't brought
> the rest of the Forrest community along with us. But now it is happening, which can
> only be good.
In two, things can only go so far. Jeff has finally brought the stuff in
"the real world (TM)" and highlighted things we did not think about.
I'm happy that you're here to work with him on this now.
What you have described is what I mean to say too. :-)
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Upayavira <uv...@upaya.co.uk>.
Nicola Ken,
> Sorry Jeff, but I don't have time nor energy to dwelve into this
> discussion further. I'm getting a bit tired about it too. Don't get me
> wrong, it's not about you, it's just that sometimes one loses interest
> in some things.
FWIW, I'm pleased that Jeff is prepared to go along with these discussions - I think
our original discussions only went so far. We got it down to one pass, and were pretty
happy - we didn't really engage further with what the real consequences of that were,
and what we potentially lost. And I think because we didn't do this, we haven't brought
the rest of the Forrest community along with us. But now it is happening, which can
only be good.
> First of all, all I want is speed and less memory usage. At least the
> same speed we are now getting with the new CLI. If any alternative
> scheme can be devised to get to comparable speed and possibly memory
> usage, I'm *completely* fine with it. IIUC What came out of the
> initial "new CLI" dicussion is that a single pass can be regarded as
> both technically and conceptually better.
I agree entirely.
> Secondly, it seems to me that you are mixing conceptual decisions with
> what are in fact just implementation "details". Things you have
> pointed out, like fixed gatherer position for example, are just fruit
> of the initial implementation, not a thorough and important design
> decision, and thus have still to be improved upon and testes in the
> real world (for us it's Forrest).
Exactly. And that is the point we are now coming to - we can move the link gathering
stage where ever we like without slowing down the CLI. In fact, having an explicit one
will speed things up, as we'll get rid of its use on cocoon: protocol pipelines.
> Finally, the new CLI is a WIP, so I applaud your effort in getting it
> better, so that it does not throw out the baby (link view) with the
> water (3 pass generation). I'm trying to see that in this process also
> the new features of the CLI (one pass gathering) are not thrown out
> themselves in the process of saving the baby ;-)
With the idea of two link gathering transformers, both of which can be placed
anywhere in a pipeline, one which extracts hrefs and xlinks (as does the current link
gatherer) and one which consumes a links namespace (which allows complete
control over your followed links, just like the links view), I think we've got the best of
both worlds. An empty bath with a baby in it :-)
Regards, Upayavira
Re: Link view goodness (Re: residuals of MIME type bug ?)
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Jeff Turner wrote, On 28/06/2003 3.59:
...
> I hope I've convinced you :) Certainly for simpler needs, hardcoding
> a LinkGathererTransformer is fine, but in general (and I hope where
> Forrest is going) we need the full power of a link view.
Sorry Jeff, but I don't have time nor energy to dwelve into this
discussion further. I'm getting a bit tired about it too.
Don't get me wrong, it's not about you, it's just that sometimes one
loses interest in some things.
So please excuse me if I won't reply to your points and I just only
present MHO.
First of all, all I want is speed and less memory usage. At least the
same speed we are now getting with the new CLI. If any alternative
scheme can be devised to get to comparable speed and possibly memory
usage, I'm *completely* fine with it.
IIUC What came out of the initial "new CLI" dicussion is that a single
pass can be regarded as both technically and conceptually better.
Secondly, it seems to me that you are mixing conceptual decisions with
what are in fact just implementation "details".
Things you have pointed out, like fixed gatherer position for example,
are just fruit of the initial implementation, not a thorough and
important design decision, and thus have still to be improved upon and
testes in the real world (for us it's Forrest).
This is the sole reason why I ask you to read that thread. It explains
the design decisions, and can help you in not necessarily
re-investigating stuff that has already been fruitfully discussed.
Finally, the new CLI is a WIP, so I applaud your effort in getting it
better, so that it does not throw out the baby (link view) with the
water (3 pass generation).
I'm trying to see that in this process also the new features of the CLI
(one pass gathering) are not thrown out themselves in the process of
saving the baby ;-)
Ciao :-)
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Link view goodness (Re: residuals of MIME type bug ?)
Posted by Jeff Turner <je...@apache.org>.
On Thu, Jun 26, 2003 at 03:08:09PM +0100, Upayavira wrote:
>
> > But I like the link-views! ;) It's one of those design elegancies
> > that makes Cocoon unique. Adding a don't-crawl-these-links option to
> > the new CLI may solve the same problem, but IMHO it's a hack in
> > comparison.
>
> What specifically is it that you like about link views? Cos at the
> moment, the alternative way of gathering links is with an invisible
> (automatically inserted) LinkGatherer transformer stage right before
> the serializer.
Conceptually, I link the link-view because:
1) Links are URIs
2) The sitemap is 100% in control of the URI space
implying:
3) The sitemap ought to be in control of link URI manipulation, not
some external cli.xconf file.
Now for practicalities:
I like the fact that the sitemap writer has full control over what is
considered a link, and what those links look like. An invisible linkgatherer
transformer effectively hardcodes:
<map:serializer name="links" src="org.apache.cocoon.serialization.LinkSerializer">
<encoding>ISO-8859-1</encoding>
</map:serializer>
<map:view name="links" from-position="last">
<map:serialize type="links"/>
</map:view>
There are various points of flexibility that the links view allows:
Alternative Link schemes
------------------------
If the user's XML doesn't happen to use XLink or @href for linking, they would
implement an alternative to LinkSerializer.
For example, imagine we want to render only PDFs. The last XSLT in
our pipeline would produce xsl:fo. The standard LinkSerializer
doesn't know about fo:external-link elements. Even if it did, we'd
want to filter out links to images, since PDFs have images inlined.
What is an image? That's up to the sitemap writer.
Encoding
--------
When serializing links in Japanese or something, wouldn't tweaking the
<encoding> tag be necessary?
Filtering unwanted links
------------------------
We can filter out unwanted links, with arbitrary precision (eg using
XPath expressions to determine what to throw out). In Forrest we use
<xsl:when test="contains(., 'api/')"> to filter out javadoc links.
Eventually, 'api/' will be determined at runtime, by querying an input
module that reads a forrest.xml config file.
I hope I've convinced you :) Certainly for simpler needs, hardcoding
a LinkGathererTransformer is fine, but in general (and I hope where
Forrest is going) we need the full power of a link view.
--Jeff
>
> Upayavira
>
Re: residuals of MIME type bug ?
Posted by Upayavira <uv...@upaya.co.uk>.
> But I like the link-views! ;) It's one of those design elegancies
> that makes Cocoon unique. Adding a don't-crawl-these-links option to
> the new CLI may solve the same problem, but IMHO it's a hack in
> comparison.
What specifically is it that you like about link views? Cos at the moment, the
alternative way of gathering links is with an invisible (automatically inserted)
LinkGatherer transformer stage right before the serializer. If that was replaced with a
LinkGatheringTransformer that you could use to gather links (and place them in a List
in the object model), but that returns the SAX events unchanged to the next stage,
you'd have your link view with a single pass. Even building in an XSLT stage that is
used before extracting the links. Just a thought.
> The argument against link-views is that it's slow: two requests
> instead of one. Isn't the correct solution to fix caching so that 99%
> of the processing between foo.html and foo.html?cocoon-view=links is
> shared? If caching worked properly, why would requesting the link
> view take much more time?
Alternatively we could try Vadim's comment on using a CachingPoint pipeline.
Thoughts?
Upayavira
Re: residuals of MIME type bug ?
Posted by Jeff Turner <je...@apache.org>.
(moving to cocoon-dev..)
On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
> There are quite a lot of new features in the Cocoon CLI that Forrest
> isn't using, for example the option to switch off mime-type checking,
> and to only scan pages once (i.e. not using the link-view) to follow
> links.
But I like the link-views! ;) It's one of those design elegancies that
makes Cocoon unique. Adding a don't-crawl-these-links option to the new
CLI may solve the same problem, but IMHO it's a hack in comparison.
The argument against link-views is that it's slow: two requests instead
of one. Isn't the correct solution to fix caching so that 99% of the
processing between foo.html and foo.html?cocoon-view=links is shared? If
caching worked properly, why would requesting the link view take much
more time?
--Jeff
...
> Regards, Upayavira
>
Re: residuals of MIME type bug ?
Posted by Jeff Turner <je...@apache.org>.
(moving to cocoon-dev..)
On Thu, Jun 26, 2003 at 08:43:12AM +0100, Upayavira wrote:
> There are quite a lot of new features in the Cocoon CLI that Forrest
> isn't using, for example the option to switch off mime-type checking,
> and to only scan pages once (i.e. not using the link-view) to follow
> links.
But I like the link-views! ;) It's one of those design elegancies that
makes Cocoon unique. Adding a don't-crawl-these-links option to the new
CLI may solve the same problem, but IMHO it's a hack in comparison.
The argument against link-views is that it's slow: two requests instead
of one. Isn't the correct solution to fix caching so that 99% of the
processing between foo.html and foo.html?cocoon-view=links is shared? If
caching worked properly, why would requesting the link view take much
more time?
--Jeff
...
> Regards, Upayavira
>
Re: unsubscribe (Was: residuals of MIME type bug ?)
Posted by David Crossley <cr...@indexgeo.com.au>.
David Crossley wrote:
> yachting heritage <co...@yachting-heritage.com> wrote:
> > how could we avoid to received your bloody mail ?
>
> I had sent direct email to this person when they first
> whined, and Nicola Ken also replied on-list. It seems
> that they made no effort. I am now trying my moderator
> powers to do it.
Woops, sorry i just saw Steven's reply from yesterday.
My bad email filters had channelled some of my forrest
email into cocoon-dev folder (with that Cc mess).
Anyway, that user has now been cast adrift.
--David
unsubscribe (Was: residuals of MIME type bug ?)
Posted by David Crossley <cr...@indexgeo.com.au>.
yachting heritage <co...@yachting-heritage.com> wrote:
> how could we avoid to received your bloody mail ?
I had sent direct email to this person when they first
whined, and Nicola Ken also replied on-list. It seems
that they made no effort. I am now trying my moderator
powers to do it.
--David
Re: residuals of MIME type bug ?
Posted by yachting heritage <co...@yachting-heritage.com>.
how could we avoid to received your bloody mail ?
----- Original Message -----
From: "Nicola Ken Barozzi" <ni...@apache.org>
To: <fo...@xml.apache.org>
Sent: Thursday, June 26, 2003 9:20 AM
Subject: Re: residuals of MIME type bug ?
>
> Upayavira wrote, On 26/06/2003 9.43:
>
> > There are quite a lot of new features in the Cocoon CLI that Forrest
isn't using, for
> > example the option to switch off mime-type checking, and to only scan
pages once
> > (i.e. not using the link-view) to follow links.
>
> We have to use them, I agree. In fact Forrest was the promary use case
> of having more speed with the new CLI and for not using the mimetype
thing.
>
> > I believe there are still some problems with these new features in the
CLI, but it
> > should be possible to fix these. [For example, links being gathered on
pipelines
> > referenced via cocoon: protocol - I've found why, but not yet fixed it].
> >
> > Is anyone interested in looking into how to upgrade Forrest to use these
new
> > features?
> >
> > I think that doing this would stand a chance of resolving all of Luc's
problems, and
> > give me some people to do some solid debugging of the CLI.
> >
> > Any takers?
>
> --
> Nicola Ken Barozzi nicolaken@apache.org
> - verba volant, scripta manent -
> (discussions get forgotten, just code remains)
> ---------------------------------------------------------------------
>
>
Re: residuals of MIME type bug ?
Posted by Nicola Ken Barozzi <ni...@apache.org>.
Upayavira wrote, On 26/06/2003 9.43:
> There are quite a lot of new features in the Cocoon CLI that Forrest isn't using, for
> example the option to switch off mime-type checking, and to only scan pages once
> (i.e. not using the link-view) to follow links.
We have to use them, I agree. In fact Forrest was the promary use case
of having more speed with the new CLI and for not using the mimetype thing.
> I believe there are still some problems with these new features in the CLI, but it
> should be possible to fix these. [For example, links being gathered on pipelines
> referenced via cocoon: protocol - I've found why, but not yet fixed it].
>
> Is anyone interested in looking into how to upgrade Forrest to use these new
> features?
>
> I think that doing this would stand a chance of resolving all of Luc's problems, and
> give me some people to do some solid debugging of the CLI.
>
> Any takers?
--
Nicola Ken Barozzi nicolaken@apache.org
- verba volant, scripta manent -
(discussions get forgotten, just code remains)
---------------------------------------------------------------------
Re: residuals of MIME type bug ?
Posted by Upayavira <uv...@upaya.co.uk>.
There are quite a lot of new features in the Cocoon CLI that Forrest isn't using, for
example the option to switch off mime-type checking, and to only scan pages once
(i.e. not using the link-view) to follow links.
I believe there are still some problems with these new features in the CLI, but it
should be possible to fix these. [For example, links being gathered on pipelines
referenced via cocoon: protocol - I've found why, but not yet fixed it].
Is anyone interested in looking into how to upgrade Forrest to use these new
features?
I think that doing this would stand a chance of resolving all of Luc's problems, and
give me some people to do some solid debugging of the CLI.
Any takers?
Regards, Upayavira
On 25 Jun 2003 at 8:28, MAISONOBE Luc wrote:
> Hello,
>
> I have updated my forrest source tree yesterday evening (european
> time) including the new cocoon jar.
>
> This has solved the file.extnull problem I had encountered like
> everyone else, and also partly solved my src/content/file.xml problems
> (those files were parsed and slightly modified despite they are not in
> the xdocs sub-directory). The files are now correctly copied
> untouched.
>
> The remaining problems I see are :
>
> - a single java source file that should also be copied as is has
> its name changed from EllipseArc.java to EllipseArc.java.txt
> (before the cocoon change, it was renamed with the .text suffix
> rather than .txt so I think this is still related to the same
> bug)
>
> - a compressed PostScript file is copied twice, once as is and
> another time in decompressed form (with the .gz suffix removed)
>
> - spurious error messages appear during site generation:
> Despite the files are copied near the beginning of forrest run, I
> get the following error messages later on (and the files are
> listed in the brokenlist.txt file). I noticed also that one of my
> xml file did generate an error message and was not listed. I did
> not understand why this one was different from the other ones.
>
> ------------------------------------------------------------------
>
> ...lots of stuff omitted ...
>
> * [0] skin/images/valid-html401.png
> * [41] downloads.html
> * [0]
> -> [broken page] software/rkcheck/HighamHall.xml <-
>
> /home/luc/spaceroots.org/forrest/build/tmp/context/content/xd
> ocs/software/rkcheck/HighamHall.xml (No such file or director y)
>
> ...lots of stuff omitted ...
>
> ------------------------------------------------------------------
>
> It seems as if due to the .xml suffix, some part of the
> process expect the file to really live in the xdocs directory,
> and logically doesn't find them here. I suspect my file names are
> recognized by two different rules, one based on their directory
> (not in xdocs), and another one based on their suffix (*.xml).
> The first rule working correctly for these file types now, and
> the other rule failing. If this is the case, a simple correction
> would be to extend the pattern to something like */xdocs/*.xml or
> similar. Does it make sense ?
>
> I can try to look at this problem by myself, could you advise me were
> to start (sitemap.xmap only ?). I don't think I can handle the first
> two problems though.
>
> Luc
>
>
>