Re: Solving matching problems caused by plugins

[Dave Brondsema <da...@brondsema.net>]

Ross Gardler wrote:
> Ross Gardler wrote:
> 
>> David Crossley wrote:
>>
> 
>>> It mostly works now, but building our own docs and building
>>> the test site shows some other related new bug. I can only
>>> describe the symptoms at this stage ...
>>
>>
>>
>>>
>>> ------ from 'forrest seed site' ...
>>>   <link message="...content/xdocs/samples/ihtml-sample.xml
>>> (No such file or directory)" uri="samples/ihtml-sample.html">
>>>
>>>   <link message="...content/xdocs/samples/wiki-sample.xml
>>> (No such file or directory)" uri="samples/wiki-sample.html">
>>> ------
>>>
>>> ... those two indicate that the special filetype tests
>>> in forrest.xmap at line 234 are not being triggered.
> 
> 
> These should really be extracted to a plugin which will solve the 
> problem. I think that the only thing that should be left in Forrest is 
> the handling of our intermediate format (this will be plural when we 
> move to XHTML2).
> 
> Dues anyone have a problem with that?
> 

I think that's the right idea.  Lots of stuff can be moved to be a 
plugin.  For example, changes/todo.html as a plugin would make that 
functionality optional and so people can use those reserveds names.

I haven't tried yet.. does the plugin location allow for input formats 
and output formats?  Is it in a location that wholesite could be in a 
plugin?

We will also have to document a "plugin contract" of what pipeline parts 
will be made available and supported.  I think we have need to work 
through some more real examples of plugins before we fully understand 
their power and limitations.  Then we can best write the contract.

-- 
Dave Brondsema : dave@brondsema.net
http://www.splike.com : programming
http://csx.calvin.edu : student org
http://www.brondsema.net : personal

Re: Solving matching problems caused by plugins

[Ross Gardler <rg...@apache.org>]

David Crossley wrote:
> Ross Gardler wrote:
> 
>>Dave Brondsema wrote:
>>
>>>Ross Gardler wrote:
>>>I think that's the right idea.  Lots of stuff can be moved to be a 
>>>plugin.  For example, changes/todo.html as a plugin would make that 
>>>functionality optional and so people can use those reserveds names.
>>>
>>>I haven't tried yet.. does the plugin location allow for input formats 
>>>and output formats?
>>
>>I'm starting to think that we need a more flexible plugin system that 
>>would allow insertion at different points in the generation process.
>>
>>For example, IMSManifest is currently broken because of having to move 
>>the plugin sitemap mount to *after* the generation of things like 
>>site.xml, faq.xml etc.
>>
>>Your question about input and output formats is another point, output 
>>formats would need to be mounted in a different location from the input 
>>formats (which is all I have worked with to date).
>>
>>I was thinking of moving to a system where plugins can provide three 
>>sitemaps:
>>
>>preprocessing (comes before *everything* in core - such sitemaps would 
>>have to be very careful to only match what they need, i.e. IMSManifest 
>>would match abs-linkmap, site.xml. tabs.xml etc.)
> 
> 
> Already the project sitemap.xmap must be careful to
> only match what they need.
> 
> So where does their project sitemap fit into this?

Yes, and I suspect that this will start to cause us real problems. As 
Dave Bronsema suggested earlier in this thread, we need to create a 
contract for plugins. I think this same contract can be used for the 
project sitemap. In the meantime we'll leave project sitemap where it is 
since we will learn more about the correct handling through working with 
the plugins.

>>input formats (these would be placed where the current mount is, i.e. 
>>just before the default **.xml match in forrest.xmap)
> 
> 
> Would a project sitemap still be able to match those
> if they needed to?

Yes, the project sitemap is (and will always be) mounted before the 
plugins. In other words, the order of processing precedence would be 
project -> plugins -> forrest.

There is the question of which plugin gets preference since there could 
be conflicts. Right now I have done two things to try and tackle this:

1) only the plugins required for the current project will be mounted, 
regardless of what plugins are on the local HD

2) plugins will be processed in the order they appear in the 
project.requried.plugins property

I believe this will be sufficient for the time being. For example, it 
will handle the overlap between the current simplified-docbook plugin 
and the proposed full-docbook plugin.

> [snip]
> 
>>>We will also have to document a "plugin contract" of what pipeline parts 
>>>will be made available and supported.  I think we have need to work 
>>>through some more real examples of plugins before we fully understand 
>>>their power and limitations.  Then we can best write the contract.
>>
>>I agree. I will, over the coming weeks move all none core functionality 
>>from core into a plugin. That should give us a fair number of use cases 
>>to at least draft the contract from.
>>
>>The question at the moment is where to *stop* moving stuff into plugins.
> 
> 
> Bit-by-bit until we can understand the core sitemaps again.
> If we start to duplicate functionality in each plugin, then
> it will become apparent where to stop.

OK sounds wise. I'll keep going until someone holds up a stop sign (I 
suspect I may miss it myself - extracting plugin code is quick reward 
stuff - much better than building flat pack furnitre ;-))

Ross

Re: Solving matching problems caused by plugins

[David Crossley <cr...@apache.org>]

Ross Gardler wrote:
> Dave Brondsema wrote:
> > Ross Gardler wrote:
> > I think that's the right idea.  Lots of stuff can be moved to be a 
> > plugin.  For example, changes/todo.html as a plugin would make that 
> > functionality optional and so people can use those reserveds names.
> > 
> > I haven't tried yet.. does the plugin location allow for input formats 
> > and output formats?
> 
> I'm starting to think that we need a more flexible plugin system that 
> would allow insertion at different points in the generation process.
> 
> For example, IMSManifest is currently broken because of having to move 
> the plugin sitemap mount to *after* the generation of things like 
> site.xml, faq.xml etc.
> 
> Your question about input and output formats is another point, output 
> formats would need to be mounted in a different location from the input 
> formats (which is all I have worked with to date).
> 
> I was thinking of moving to a system where plugins can provide three 
> sitemaps:
> 
> preprocessing (comes before *everything* in core - such sitemaps would 
> have to be very careful to only match what they need, i.e. IMSManifest 
> would match abs-linkmap, site.xml. tabs.xml etc.)

Already the project sitemap.xmap must be careful to
only match what they need.

So where does their project sitemap fit into this?

> input formats (these would be placed where the current mount is, i.e. 
> just before the default **.xml match in forrest.xmap)

Would a project sitemap still be able to match those
if they needed to?

[snip]
> > We will also have to document a "plugin contract" of what pipeline parts 
> > will be made available and supported.  I think we have need to work 
> > through some more real examples of plugins before we fully understand 
> > their power and limitations.  Then we can best write the contract.
> 
> I agree. I will, over the coming weeks move all none core functionality 
> from core into a plugin. That should give us a fair number of use cases 
> to at least draft the contract from.
> 
> The question at the moment is where to *stop* moving stuff into plugins.

Bit-by-bit until we can understand the core sitemaps again.
If we start to duplicate functionality in each plugin, then
it will become apparent where to stop.

> I'm starting to think that Forrest core should follow the Eclipse model 
> where the isn't any real functionality in core, just a solid framework 
> onto which to build the required functionality.

That sounds sensible.

> Our releases would then become (for example)

How we do releases and what they contain is something
that we need to carefully consider. Let us first get our
plugin framework settled a bit.

> - Forrest for Software Projects (core + site.xml + HTML + PDF + JavaDoc)
> 
> - Forrest for Learning Objects (core + IMSManifest + OpenOffice + Latex 
> +PDF)
> 
> - Forrest for Technical Authors (core + site.xml + docbook + PDF)
> 
> Etc.
> 
> I may be getting a little ahead of myself though ;-)

No, i think that you are correct to look ahead.

-- 
David Crossley

Re: Solving matching problems caused by plugins

[Ross Gardler <rg...@apache.org>]

Dave Brondsema wrote:
> Ross Gardler wrote:
> I think that's the right idea.  Lots of stuff can be moved to be a 
> plugin.  For example, changes/todo.html as a plugin would make that 
> functionality optional and so people can use those reserveds names.
> 
> I haven't tried yet.. does the plugin location allow for input formats 
> and output formats?

I'm starting to think that we need a more flexible plugin system that 
would allow insertion at different points in the generation process.

For example, IMSManifest is currently broken because of having to move 
the plugin sitemap mount to *after* the generation of things like 
site.xml, faq.xml etc.

Your question about input and output formats is another point, output 
formats would need to be mounted in a different location from the input 
formats (which is all I have worked with to date).

I was thinking of moving to a system where plugins can provide three 
sitemaps:

preprocessing (comes before *everything* in core - such sitemaps would 
have to be very careful to only match what they need, i.e. IMSManifest 
would match abs-linkmap, site.xml. tabs.xml etc.)

input formats (these would be placed where the current mount is, i.e. 
just before the default **.xml match in forrest.xmap)

output formats (not sure where yet, as I haven't thought through a use 
case, I suspect Sean's Docbook plugin will be the first)

>  Is it in a location that wholesite could be in a 
> plugin?

No idea. To be honest, I've not looked since this is currently broken 
anyway.

> We will also have to document a "plugin contract" of what pipeline parts 
> will be made available and supported.  I think we have need to work 
> through some more real examples of plugins before we fully understand 
> their power and limitations.  Then we can best write the contract.

I agree. I will, over the coming weeks move all none core functionality 
from core into a plugin. That should give us a fair number of use cases 
to at least draft the contract from.

The question at the moment is where to *stop* moving stuff into plugins.

I'm starting to think that Forrest core should follow the Eclipse model 
where the isn't any real functionality in core, just a solid framework 
onto which to build the required functionality. Our releases would then 
become (for example)

- Forrest for Software Projects (core + site.xml + HTML + PDF + JavaDoc)

- Forrest for Learning Objects (core + IMSManifest + OpenOffice + Latex 
+PDF)

- Forrest for Technical Authors (core + site.xml + docbook + PDF)

Etc.

I may be getting a little ahead of myself though ;-)

Ross