Re: [RT] A new Forrest implementation?

[Ross Gardler <rg...@apache.org>]

Ross Gardler wrote:
> This is a Random Thought. The ideas contained within are not fully
> developed and are bound to have lots of holes. The idea is to promote
> healthy discussion, so please, everyone, dive in and discuss.
> 
> The Problem
> ===========
> 
> Forrest is built on Cocoon, a web application framework, but "all" it
> does is XML publishing. This means we have a monolithic web application
> framework that is doing nothing more than managing a processing pipeline
> and doing XSLT transformations.

There was a fair amount of discussion around this, with very little
agreement but plenty of constructive ideas. During the course of 
discussions I promised to put together a basic implementation of what I 
perceive to be Forrest 2.0

I knocked something together a couple of days later. This was very
basic, but did what I needed it to do. Since then I have been intending 
to find the time to finish it off and create more examples. 
Unfortunately, I have been unable to find that time.

The main blocker right now is that it currently does what *I* need it
to do, but to be useful in the wider community it needs to do much more 
(see below).

The second blocker is that I've been waiting for Forrest 0.8 to reach a 
release state, however, it seems Forrest is in a slow period, perhaps 
waiting is the wrong thing to do?

So, since I now need to do more work on this code I thought I'd see if 
there is any interest in the work within the Forrest Community. Note, 
I'm keeping the Random Thoughts thread, this is a radical change in many 
things that Forrest does and should only be considered as a discussion 
point. I'm not asking the Forrest team to endorse this code (although 
would welcome such endorsement). I'm really asking if the Forrest team 
would like to see the work as it progresses in the whiteboard (I will 
not be using Forrest 0.8 on this project, for all the reasons described 
in this RT).

Copied below is the first part of the readme for this code...

--------------------
Start of copied text
--------------------

NOTE: this document is written as though this is approved as a Forrest 2
implementation, however, it is important to recognise that this is only 
an experiment at this time. This code is not currently adopted by the 
Apache Forrest project and is not available via any public distribution 
mechanism at this time.

Forrest 2.0
===========

Forrest2 is a complete rewrite of the XML publishing framework Apache
Forrest.

It draws on the lessons learned during development and use of earlier
versions of Forrest, but it takes the opportunity to simplify
implementation and use wherever possible.

Forrest2 is no longer built upon Apache Cocoon, it is a plain Java
application.

This may prompt you to ask "does this mean we lose the power of the
Apache Cocoon framework?" The answer to that question is an emphatic no.
You see, Apache Cocoon can still be used by Forrest, but you are no
longer tied to it as was the case with earlier versions of Forrest.

Breaking away from Cocoon means that embedding Forrest2 in another
application is much easier. In fact, since the new Forrest can be
packaged as a Cocoon generator it is, ironically enough, easier to use
in another Cocoon application than the original Forrest was. At the same
time it is easier to embed Forrest in any other type of application.

The core of Forrest 2 is only a few megabytes in size (and can doubtless
be trimmed even further with work). This compares very favorably with 
40Mb+ for Cocoon based versions of Forrest.

Furthermore, the removal of Cocoon as the core framework means that
extending Forrest no longer requires an understanding of Cocoon, in
fact, Forrest2 can be extended very easily with just about any
technology (including Cocoon). It is now possible to extend Forrest in
any way you desire, all you need to do is expose your services via a
REST style API or a Java API.

Of course, it's easier to use too...

---------------
End copied text
---------------

[At this point the readme goes on to describe some *simple* use cases 
that illustrate the use of Forest Core and some example plugins - if it 
helps or anyone wants it I'll post the full readme]

This code is very incomplete when compared to Forrest 0.8. It needs 
quite a bit of work, but I believe it provides a means to address all 
the problems I identified in this RT. Of course, it does not address 
documentation issues raised, in fact it would make most of our current 
docs obsolete!

It is also worth noting that many of the "grand claims" in the above 
readme snippet are theoretical only. At this time Forrest2 provides:

- a Spring IoC framework for connecting XML processing plugins

- a few simple examples of input plugins

- a couple of output plugin examples

- a basic locationmap implementation

- a live commercial app that uses a number of input plugins to handle a 
number of proprietary XML formats (i.e. it works in real life too)

- a webapp for dynamic creation of documents

What it doesn't implement yet:

- effective caching

- "useful" aggregation (it does implement aggregation but it is not 
flexible enough to be of any real use outside our current use case)

- streaming of processing pipeline data

- static site generation (just needs a crawler)

- pattern matching in the locationmap (need to put RE evaluation in 
there - currently just does string matching)

- a Cocoon generator as suggested in this RT (should be really simple 
for anyone who needs it, there is a servlet showing how to use the Java API)

- view processing (i.e. it doesn't generate complete web pages, this is, 
in my opinion, outside the scope of Forrest core. Currently we have a 
Wicket application that creates the view, but the creation of a Cocoon 
generator would open up many possibilities to make existing Forrest devs 
comfortable)

What are the main benefits over current Forrest?

- see read me intro above (i.e. embedding forrest and size)

- much simpler configuration (all done in the Spring config files)

- minimal XML "programming" (i.e. no complex sitemap to define 
processing paths - this needs exploration and discussion)

- a hell of a lot faster (although this is anecdotal rather than measured)

When discussing this implementation move in the past I have felt 
considerable resistance to these implementation ideas. This may have 
been my inability to describe things clearly (the code will help here) 
but it may have been because people do not have the need for the 
improvements this brings and therefore don't want to take such a radical 
step.

So, shall I conduct this work in whiteboard? Do you have more questions 
first? Would putting it in whiteboard be harmful to Apache Forrest 0.8?

???

Ross

Re: [RT] A new Forrest implementation?

[Ross Gardler <rg...@apache.org>]

Tim Williams wrote:
> On 11/13/06, Ross Gardler <rg...@apache.org> wrote:
>> Ross Gardler wrote:

...

>> So, shall I conduct this work in whiteboard? Do you have more questions
>> first? Would putting it in whiteboard be harmful to Apache Forrest 0.8?
> 

...

> Anyway, I'd like to see it in the whiteboard if for no other reason
> than to have something concrete to frame this continued discussion
> around.

Yep, that's my intention.

>  I think it won't harm 0.8 at all.

Normally I would put it straight in and see what happens, my concern is 
with diverting important attention away from finishing 0.8. We all have 
a limited time and shiny new things are often more interesting than 
polishing the old (but still useful) things.

Anyway, I'll go with lazy consensus - so if people think it may be a bad 
move please say so. I'm happy to work on it in private and come back 
with more complete code after 0.8 release.

Ross

Re: [RT] A new Forrest implementation?

[Tim Williams <wi...@gmail.com>]

On 11/13/06, Ross Gardler <rg...@apache.org> wrote:
> Ross Gardler wrote:
> > This is a Random Thought. The ideas contained within are not fully
> > developed and are bound to have lots of holes. The idea is to promote
> > healthy discussion, so please, everyone, dive in and discuss.
> >
> > The Problem
> > ===========
> >
> > Forrest is built on Cocoon, a web application framework, but "all" it
> > does is XML publishing. This means we have a monolithic web application
> > framework that is doing nothing more than managing a processing pipeline
> > and doing XSLT transformations.
>
> There was a fair amount of discussion around this, with very little
> agreement but plenty of constructive ideas. During the course of
> discussions I promised to put together a basic implementation of what I
> perceive to be Forrest 2.0
>
> I knocked something together a couple of days later. This was very
> basic, but did what I needed it to do. Since then I have been intending
> to find the time to finish it off and create more examples.
> Unfortunately, I have been unable to find that time.
>
> The main blocker right now is that it currently does what *I* need it
> to do, but to be useful in the wider community it needs to do much more
> (see below).
>
> The second blocker is that I've been waiting for Forrest 0.8 to reach a
> release state, however, it seems Forrest is in a slow period, perhaps
> waiting is the wrong thing to do?
>
> So, since I now need to do more work on this code I thought I'd see if
> there is any interest in the work within the Forrest Community. Note,
> I'm keeping the Random Thoughts thread, this is a radical change in many
> things that Forrest does and should only be considered as a discussion
> point. I'm not asking the Forrest team to endorse this code (although
> would welcome such endorsement). I'm really asking if the Forrest team
> would like to see the work as it progresses in the whiteboard (I will
> not be using Forrest 0.8 on this project, for all the reasons described
> in this RT).
>
> Copied below is the first part of the readme for this code...
>
> --------------------
> Start of copied text
> --------------------
>
> NOTE: this document is written as though this is approved as a Forrest 2
> implementation, however, it is important to recognise that this is only
> an experiment at this time. This code is not currently adopted by the
> Apache Forrest project and is not available via any public distribution
> mechanism at this time.
>
> Forrest 2.0
> ===========
>
> Forrest2 is a complete rewrite of the XML publishing framework Apache
> Forrest.
>
> It draws on the lessons learned during development and use of earlier
> versions of Forrest, but it takes the opportunity to simplify
> implementation and use wherever possible.
>
> Forrest2 is no longer built upon Apache Cocoon, it is a plain Java
> application.
>
> This may prompt you to ask "does this mean we lose the power of the
> Apache Cocoon framework?" The answer to that question is an emphatic no.
> You see, Apache Cocoon can still be used by Forrest, but you are no
> longer tied to it as was the case with earlier versions of Forrest.
>
> Breaking away from Cocoon means that embedding Forrest2 in another
> application is much easier. In fact, since the new Forrest can be
> packaged as a Cocoon generator it is, ironically enough, easier to use
> in another Cocoon application than the original Forrest was. At the same
> time it is easier to embed Forrest in any other type of application.
>
> The core of Forrest 2 is only a few megabytes in size (and can doubtless
> be trimmed even further with work). This compares very favorably with
> 40Mb+ for Cocoon based versions of Forrest.
>
> Furthermore, the removal of Cocoon as the core framework means that
> extending Forrest no longer requires an understanding of Cocoon, in
> fact, Forrest2 can be extended very easily with just about any
> technology (including Cocoon). It is now possible to extend Forrest in
> any way you desire, all you need to do is expose your services via a
> REST style API or a Java API.
>
> Of course, it's easier to use too...
>
> ---------------
> End copied text
> ---------------
>
> [At this point the readme goes on to describe some *simple* use cases
> that illustrate the use of Forest Core and some example plugins - if it
> helps or anyone wants it I'll post the full readme]
>
> This code is very incomplete when compared to Forrest 0.8. It needs
> quite a bit of work, but I believe it provides a means to address all
> the problems I identified in this RT. Of course, it does not address
> documentation issues raised, in fact it would make most of our current
> docs obsolete!
>
> It is also worth noting that many of the "grand claims" in the above
> readme snippet are theoretical only. At this time Forrest2 provides:
>
> - a Spring IoC framework for connecting XML processing plugins
>
> - a few simple examples of input plugins
>
> - a couple of output plugin examples
>
> - a basic locationmap implementation
>
> - a live commercial app that uses a number of input plugins to handle a
> number of proprietary XML formats (i.e. it works in real life too)
>
> - a webapp for dynamic creation of documents
>
> What it doesn't implement yet:
>
> - effective caching
>
> - "useful" aggregation (it does implement aggregation but it is not
> flexible enough to be of any real use outside our current use case)
>
> - streaming of processing pipeline data
>
> - static site generation (just needs a crawler)
>
> - pattern matching in the locationmap (need to put RE evaluation in
> there - currently just does string matching)
>
> - a Cocoon generator as suggested in this RT (should be really simple
> for anyone who needs it, there is a servlet showing how to use the Java API)
>
> - view processing (i.e. it doesn't generate complete web pages, this is,
> in my opinion, outside the scope of Forrest core. Currently we have a
> Wicket application that creates the view, but the creation of a Cocoon
> generator would open up many possibilities to make existing Forrest devs
> comfortable)
>
> What are the main benefits over current Forrest?
>
> - see read me intro above (i.e. embedding forrest and size)
>
> - much simpler configuration (all done in the Spring config files)
>
> - minimal XML "programming" (i.e. no complex sitemap to define
> processing paths - this needs exploration and discussion)
>
> - a hell of a lot faster (although this is anecdotal rather than measured)
>
> When discussing this implementation move in the past I have felt
> considerable resistance to these implementation ideas. This may have
> been my inability to describe things clearly (the code will help here)
> but it may have been because people do not have the need for the
> improvements this brings and therefore don't want to take such a radical
> step.
>
> So, shall I conduct this work in whiteboard? Do you have more questions
> first? Would putting it in whiteboard be harmful to Apache Forrest 0.8?

I think we're in a bit of a catch-22 here.  I personally haven't been
able to fully appreciate what you're talking about until I see the
code, but I'm also not willing to commit to anything (not that you're
asking) until I fully understand what you're talking about.

Anyway, I'd like to see it in the whiteboard if for no other reason
than to have something concrete to frame this continued discussion
around.  I think it won't harm 0.8 at all.

Thanks,
--tim