Include mechanism for forrest...

[Erik Brakkee <er...@brakkee.org>]

Hi,


I have made a simple extension to Forrest for including files. It is 
based on the existing xinclude support in Forrest but is
a bit more user friendly. Detailed motiviation and instructions can be 
found at

   
http://wamblee.org/forrest-extensions/index.html#Download+and+installation

Right now it is implemented through the project sitemap.xmap file, but 
perhaps an include mechanism like this could be provided
by a future version of forrest out of the box.

Cheers
  Erik

Re: Include mechanism for forrest...

[Ross Gardler <rg...@apache.org>]

Erik Brakkee wrote:
> As I see it it just revolves around usability issues. See below.

OK, I'll reread and summarise from a user perspective...

> I have recently introduced Forrest at my work to generate documentation 
> and it is really a big hit, with especially the include feature being 
> really useful. Some of our documents tend to get big and splitting them 
> up makes it so much easier. If I would force my users to write the 
> <xi:include> markup above, I would have had a great problem getting it 
> accepted.

I'm still not convinced of the whole value of this extension, but I am 
convinced of some of the usability issues you raise. This makes the 
extension valuable in some use cases.

My biggest concern is that it is yet another processing step that every 
single document must go through before being output. This is going to 
have quite an impact on our performance, something we are taking more 
notice of as we get closer to a 1.0 release.

 From a usability perspective the automatic fallback is not a good thing 
in my opinion. Most users do not want Forrest to hide the fact that 
there is a problem with their generated site. Quite the reverse, they 
rely on Forrest to validate that all content is present and correct. 
However, I guess this could be made configurable:

<include src="myfile.xml" fallback="false"/>

The easier specification of what is to be included is nice and I see 
extensions to it, for example, include only a section:

<include src="myfile.xml" section="sectionID"/>

Not requiring paths relative to root is a great enhancement of current 
functionality.

However...

These advantages only apply to the small number of users wishing to 
include external documents I don't think these advantages are worth the 
performance hit we would get if we put it into core. That being said, I 
think this is a great candidate for an internal plugin. This will make 
it much easier for users who do lots of inclusion work to get this 
functionality, whilst leaving those concerned about performance to use 
the existing XInclude functionality.

I'm +1 on this becoming an internal plugin, so that it can be used by 
those who need it, but -0 on it being a core part of Forrest.

WDOT?

Ross

Re: Include mechanism for forrest...

[David Crossley <cr...@apache.org>]

Erik Brakkee wrote:
> David Crossley wrote:
> 
> >It seems to me that this proposal is missing the
> >abilities of the Cocoon sitemap and is just treating
> >Cocoon as an xml processor.
>
> The include mechanism is using the features of the sitemap. It is 
> nothing more than a layer to make xincludes in Forrest more usable.
> 
> Right now, since the mechanism  is implemented on top of forrest and not 
> inside of forrest, there is some duplication of work so that
> new schemes added to the site map would also have to be added to the 
> include mechanism. Nevertheless, that would not be necessary if it is 
> implemented inside Forrest.

I meant that it is often better to use map:aggregate and
get the parts from cocoon: pipelines that trim them down
to exactly the required information, rather than using xinclude.

-David

Re: Include mechanism for forrest...

[Erik Brakkee <er...@brakkee.org>]

Hi,

David Crossley wrote:

>It seems to me that this proposal is missing the
>abilities of the Cocoon sitemap and is just treating
>Cocoon as an xml processor.
>  
>
The include mechanism is using the features of the sitemap. It is 
nothing more than a layer to make xincludes in Forrest more usable.

Right now, since the mechanism  is implemented on top of forrest and not 
inside of forrest, there is some duplication of work so that
new schemes added to the site map would also have to be added to the 
include mechanism. Nevertheless, that would not be necessary if it is 
implemented inside Forrest.

>Also Cocoon will cache the results of each pipeline, so if
>the xincluded file changes, then the changes will have
>no effect.
>  
>
I don't understand what you mean here. When the included files changes, 
the changes definitely come through. Besides, if this problem would 
exist with this mechanism, then it would also exist with the xinclude 
mechanism which is already inside forrest.

>-David
>  
>
Perhaps it would be good if any of you tried it out and had a closer 
look to see how it actuall works. A good starting point is to do a 
'forrest seed' and then to simply copy the files that are available for 
download.

Cheers
  Erik

Re: Include mechanism for forrest...

[David Crossley <cr...@apache.org>]

Ross Gardler wrote:
> David Crossley wrote:
> >Also Cocoon will cache the results of each pipeline, so if
> >the xincluded file changes, then the changes will have
> >no effect.
> 
> The XInclude transformer is not cacheable. This is good with respect to 
> the problem you identify, but bad with respect to performance issues, 
> since the include is processed regardless of whether the included 
> document is changed, thus the generated document can never be cached.

Ah yeah, sorry. I had things the wrong way around.

So it is not a practice that we want to encourage,
though if people want to do it, then they should be
able to. Akin to using the document() function.

-David

> For simple includes we should use CInclude instead, this is cacheable, 
> however, CInclude does not allow the use of XPath so does not fit this 
> use case.
> 
> Ross

Re: Include mechanism for forrest...

[Ross Gardler <rg...@apache.org>]

David Crossley wrote:
> Also Cocoon will cache the results of each pipeline, so if
> the xincluded file changes, then the changes will have
> no effect.

The XInclude transformer is not cacheable. This is good with respect to 
the problem you identify, but bad with respect to performance issues, 
since the include is processed regardless of whether the included 
document is changed, thus the generated document can never be cached.

For simple includes we should use CInclude instead, this is cacheable, 
however, CInclude does not allow the use of XPath so does not fit this 
use case.

Ross

Re: Include mechanism for forrest...

[David Crossley <cr...@apache.org>]

It seems to me that this proposal is missing the
abilities of the Cocoon sitemap and is just treating
Cocoon as an xml processor.

Also Cocoon will cache the results of each pipeline, so if
the xincluded file changes, then the changes will have
no effect.

-David

Re: Include mechanism for forrest...

[Erik Brakkee <er...@brakkee.org>]

Hi Ross,

As I see it it just revolves around usability issues. See below.

Ross Gardler wrote:

> I don't actually see the need for this, your justification for using 
> this instead of XInclude is:
>
> 1  Adding XInclude elements to your document breaks the DTD so that 
> you can no longer validate your documents.
> 2 XInclude requires either full path names of files or files relative 
> to the root of the site. Unfortunately, the existing link rewriting in 
> Forrest generates links relative to the current file.
> 3 Typically you do not want to include the full contents of anothert 
> Xdoc document but only the content of the body tag. Using Xinclude 
> directly requires an author to write non-trivial XML with xpointer 
> expressions.
>
> Let me address each point:
>
> 1) Since the XInclude elements are in a different namespace you can 
> still validate your documents

If you add an element from another namespace to your document, you still 
must modify your DTD to accept this element. I still have to import 
other DTD declarations (found on the Forrest site) to avoid validation 
errors. I find this approach a bit cumbersome.  It would be better if 
the document type would support includes out of the box.

>
> 2) XInclude allows you to use any legal protocol in defining your URL. 
> This means you can use cocoon:, site: and ext: in 0.7 and can ad lm: 
> to that list in 0.8-dev Therefore you don't even need to know the name 
> of the file, just how to access it. Since this information is already 
> located somewhere in your site (for example, in site.xml) then you 
> have no problem forming a URL that is usable.

XInclude is used by the include extension. Nevertheless, direct use of 
XInclude is cumbersome. For instance, you always need to add a fallback 
element yourself, which the include extension does automatically, and 
you can work with references in the same way you do elsewhere. It should 
not be difficult to make sure that every protocol supported by XInclude 
is also supported by the include extension.

In my case, site: URLs did not work with XInclude because, as I recall, 
Xinclude processing takes place before link rewriting. Also, forcing the 
user to provide URLs relative to the site root is problematic. Sometimes 
you just want to include content from the same directory without 
defining all documents in the site.xml or referring to them with URLs 
relative to the site root.

>
> 3) XInclude allows you to select parts of a document using XPath 
> statements. This means you can include any part of the document you 
> can specifcy with an XPath expression.

I am using xinclude under the covers (of course) and I really believe 
that any include facility should be based on XInclude. But again, it is 
a usability problem. If a user includes a document  file.html into his 
document from the same directory as the document, then I
want to include the content from the xdoc file.xml and specifically the 
contents of the body tag. With the include extension, you simply write

    <include file="file.html/>

With XInclude you would have to write

    <xi:include href="cocoon:/pathtofile/file.xml#xpointer(//body/*)>
        <xi:fallback>
            <p>link to file.html not found </p>
        </xi:fallback>
    </xi:include>

Also, the xpointer support in Forrest is a bit minimal. You cannot 
specify the pointer attribute to xi:include (this is ignored), and it 
does not support full xpointer syntax, such as 'range-inside(body)'. 
Using an include tag allows you to hide dependencies on the specifics of 
the underlying xinclude processor.

I have recently introduced Forrest at my work to generate documentation 
and it is really a big hit, with especially the include feature being 
really useful. Some of our documents tend to get big and splitting them 
up makes it so much easier. If I would force my users to write the 
<xi:include> markup above, I would have had a great problem getting it 
accepted.

>
> see 
> http://cocoon.apache.org/2.1/userdocs/transformers/xinclude-transformer.html 
> for more information on the XInclude transformer.
>
> Am I misunderstanding the problem your extension is addressing, or do 
> we just need to document the XInclude features better?

It would definitely help to document the XInclude feature better, but I 
still think that from a usability point of view, a simple syntax should 
be provided for includes as part of the Xdoc DTD.. I think this is also 
in the spirit of forrest.

 From the home page of forrest:
/"Forrest is designed with the new user in mind.
Forrest's focus on low "startup cost" makes it ideal for rapid 
development of small sites, where time and budget constraints do not 
allow time-wasting HTML experiments. Of course, that same methodology 
can scale up to large projects. Your development team does not need Java 
experience, or even XML skills, to use Forrest. The framework lets you 
concentrate on content and design. "/

In other words, I think this is a good reason to add an include feature 
to Forrest.

Cheers
  Erik

>
> Ross

Re: Include mechanism for forrest...

[Ross Gardler <rg...@apache.org>]

Ross Gardler wrote:
> Erik Brakkee wrote:
> 
>> Hi,
>>
>>
>> I have made a simple extension to Forrest for including files. It is 
>> based on the existing xinclude support in Forrest but is
>> a bit more user friendly. Detailed motiviation and instructions can be 
>> found at
>>
>>   
>> http://wamblee.org/forrest-extensions/index.html#Download+and+installation 
>>
>>
>> Right now it is implemented through the project sitemap.xmap file, but 
>> perhaps an include mechanism like this could be provided
>> by a future version of forrest out of the box.
> 
> 
> I don't actually see the need for this, your justification for using 
> this instead of XInclude is:
> 
> 1  Adding XInclude elements to your document breaks the DTD so that you 
> can no longer validate your documents.

...

> Let me address each point:
> 
> 1) Since the XInclude elements are in a different namespace you can 
> still validate your documents
> 

Ermmm.. obvioulsy that should read that it can be validated if we allow 
the XInclude elements and namespace in our DTD.

Pehaps we should consider making this change in our DTD, what do people 
think?

Ross

Re: Include mechanism for forrest...

[Ross Gardler <rg...@apache.org>]

Erik Brakkee wrote:
> Hi,
> 
> 
> I have made a simple extension to Forrest for including files. It is 
> based on the existing xinclude support in Forrest but is
> a bit more user friendly. Detailed motiviation and instructions can be 
> found at
> 
>   
> http://wamblee.org/forrest-extensions/index.html#Download+and+installation
> 
> Right now it is implemented through the project sitemap.xmap file, but 
> perhaps an include mechanism like this could be provided
> by a future version of forrest out of the box.

I don't actually see the need for this, your justification for using 
this instead of XInclude is:

1  Adding XInclude elements to your document breaks the DTD so that you 
can no longer validate your documents.
2 XInclude requires either full path names of files or files relative to 
the root of the site. Unfortunately, the existing link rewriting in 
Forrest generates links relative to the current file.
3 Typically you do not want to include the full contents of anothert 
Xdoc document but only the content of the body tag. Using Xinclude 
directly requires an author to write non-trivial XML with xpointer 
expressions.

Let me address each point:

1) Since the XInclude elements are in a different namespace you can 
still validate your documents

2) XInclude allows you to use any legal protocol in defining your URL. 
This means you can use cocoon:, site: and ext: in 0.7 and can ad lm: to 
that list in 0.8-dev Therefore you don't even need to know the name of 
the file, just how to access it. Since this information is already 
located somewhere in your site (for example, in site.xml) then you have 
no problem forming a URL that is usable.

3) XInclude allows you to select parts of a document using XPath 
statements. This means you can include any part of the document you can 
specifcy with an XPath expression.

see 
http://cocoon.apache.org/2.1/userdocs/transformers/xinclude-transformer.html 
for more information on the XInclude transformer.

Am I misunderstanding the problem your extension is addressing, or do we 
just need to document the XInclude features better?

Ross