[C2][Important] Content aggregation again. Resolution needed.

[Paul Russell <pa...@luminas.co.uk>]

Hi all,

There have been a few posts about content aggregation, most of which
point to previous discussion on the subject. We need to actually
*discuss* the implementation of this and make sure we get it right.
Origionally, Stefano's proposal was to use URLs to refer to the items to
aggregate, but I am not sure this is the best way to go, as in my
opinion it makes things complicated to administer and set up.

In my previous post, I mentioned two posibilities for the sitemap markup
and semantics, and having thought about it further, there are some
complexities to even these. Also, how should we handle binary objects
(images, flash objects, applets etc) in the aggregation, or should they
be left as a separate issue, since they are frequently either a graphic
design issue, or so intimately related to the content that they appear
inside one of the aggregated parts?

For example, should we be able to do the following?

<map:sitemap>
 <map:pipelines>
  <map:pipeline>
   <!-- match everything -->
   <map:match pattern="**">
    <map:aggregate>
     <map:part name="menu">
      <map:generate src="menus/default.xml"/>
     </map:part>
     <map:part name="status">
      <map:generate type="serverpages" src="content/system-status.xml"/>
     </map:part>
     <map:part name="content">
      <!-- generate individual sections -->
      <map:match pattern="news/**">
       <map:generate type="serverpages" src="content/news.xml"/>
      </map:match>
      <map:match pattern="customers/**">
       <map:generate type="serverpages" src="content/customers.xml"/>
      </map:match>
      <map:match pattern="wiki/**">
       <map:generate type="serverpages" src="content/wiki.xml"/>
      </map:match>
     </map:part>
    </map:aggregate>
    <map:translate src="stylesheets/layout.xsl"/>
    <map:translate src="stylesheets/format.xsl"/>
    <map:serialize/>
   </map:match>
  </map:pipeline>
 </map:pipelines>
</map:sitemap>

For the time being, I'll assume that we *don't* need this functionality,
but I'll try and leave it as an option. I really *really* need some
input from you guys about what you'll be using sitemap based aggregation
for, and how you envisage it working. The solution will only be as good
as the input I recieve.

I ask again: input please from committers and users alike!


Paul
-- 
Paul Russell                                 Email:   paul@luminas.co.uk
Technical Director                             Tel:  +44 (0)20 8553 6622
Luminas Internet Applications                  Fax:  +44 (0)870 28 47489
This is not an official statement or order.    Web:    www.luminas.co.uk

Re: [C2][Important] Content aggregation again. Resolution needed.

[Paul Russell <pa...@luminas.co.uk>]

* Ross Burton (ross.burton@mail.com) wrote :
> Paul Russell wrote:
> 
> > There have been a few posts about content aggregation, most of which
> > point to previous discussion on the subject. We need to actually
> > *discuss* the implementation of this and make sure we get it right.
> > Origionally, Stefano's proposal was to use URLs to refer to the items to
> > aggregate, but I am not sure this is the best way to go, as in my
> > opinion it makes things complicated to administer and set up.
> URLs are a bad idea IMHO, unless they all use a custom protocol cocoon:
> which hooks directly into the sitemap again. These cocoon: URLs should
> access either defined pipelines, or resources.

My feelings exactly. I don't like 'hacking' the Java URL infrastructure
either, if I can avoid it. I'm not sure how useful using named pipelines
is going to be -- suspect we'll only know when we start using
aggregation in earnest.

> > In my previous post, I mentioned two posibilities for the sitemap markup
> > and semantics, and having thought about it further, there are some
> > complexities to even these. Also, how should we handle binary objects
> > (images, flash objects, applets etc) in the aggregation, or should they
> > be left as a separate issue, since they are frequently either a graphic
> > design issue, or so intimately related to the content that they appear
> > inside one of the aggregated parts?
> I can't see how binary files can be part of a SAX stream, or am I
> missing something??

Stefano mentioned passing links to binary objects through the
aggregation stream, which makes sense, I *think*. Not sure if it's
something we should be aiming for first time around.

> > For the time being, I'll assume that we *don't* need this functionality,
> > but I'll try and leave it as an option. I really *really* need some
> > input from you guys about what you'll be using sitemap based aggregation
> > for, and how you envisage it working. The solution will only be as good
> > as the input I recieve.
> Cool example, I like it! Being able to have sub-pipelines in the
> map:part elements might be a bit of an overkill, but the ability to have
> transforms inside the part elements might be usefull.  If the generated
> page is going to be huge, running small transforms over the content
> before agregation may be faster than running multiple transforms over
> the agregated content to do the same thing.

Yeah, I was pondering the same thing. I don't really want to run
*everything* through 15 transforms at the end, when most of the content
streams don't need transformation.

> What are you current plans for the implementation?

I'm still looking at it at the moment (was intending to start proper
this evening, but got distracted by an exploding computer and a candle
that won't burn. what fun). I suspect that in view of your thoughts
above, it's going to be easiest to build sub pipelines for each
aggregate part, the sub pipeline will feed an AggregatorGenerator (name
tbc ;), which will be used to feed the rest of the pipeline. Sound
reasonable?



Paul

-- 
Paul Russell                                 Email:   paul@luminas.co.uk
Technical Director                             Tel:  +44 (0)20 8553 6622
Luminas Internet Applications                  Fax:  +44 (0)870 28 47489
This is not an official statement or order.    Web:    www.luminas.co.uk

Re: [C2][Important] Content aggregation again. Resolution needed.

[Ross Burton <ro...@mail.com>]

Paul Russell wrote:

> There have been a few posts about content aggregation, most of which
> point to previous discussion on the subject. We need to actually
> *discuss* the implementation of this and make sure we get it right.
> Origionally, Stefano's proposal was to use URLs to refer to the items to
> aggregate, but I am not sure this is the best way to go, as in my
> opinion it makes things complicated to administer and set up.

URLs are a bad idea IMHO, unless they all use a custom protocol cocoon:
which hooks directly into the sitemap again. These cocoon: URLs should
access either defined pipelines, or resources.

> In my previous post, I mentioned two posibilities for the sitemap markup
> and semantics, and having thought about it further, there are some
> complexities to even these. Also, how should we handle binary objects
> (images, flash objects, applets etc) in the aggregation, or should they
> be left as a separate issue, since they are frequently either a graphic
> design issue, or so intimately related to the content that they appear
> inside one of the aggregated parts?

I can't see how binary files can be part of a SAX stream, or am I
missing something??


> For example, should we be able to do the following?
> 
> <map:sitemap>
>  <map:pipelines>
>   <map:pipeline>
>    <!-- match everything -->
>    <map:match pattern="**">
>     <map:aggregate>
>      <map:part name="menu">
>       <map:generate src="menus/default.xml"/>
>      </map:part>
>      <map:part name="status">
>       <map:generate type="serverpages" src="content/system-status.xml"/>
>      </map:part>
>      <map:part name="content">
>       <!-- generate individual sections -->
>       <map:match pattern="news/**">
>        <map:generate type="serverpages" src="content/news.xml"/>
>       </map:match>
>       <map:match pattern="customers/**">
>        <map:generate type="serverpages" src="content/customers.xml"/>
>       </map:match>
>       <map:match pattern="wiki/**">
>        <map:generate type="serverpages" src="content/wiki.xml"/>
>       </map:match>
>      </map:part>
>     </map:aggregate>
>     <map:translate src="stylesheets/layout.xsl"/>
>     <map:translate src="stylesheets/format.xsl"/>
>     <map:serialize/>
>    </map:match>
>   </map:pipeline>
>  </map:pipelines>
> </map:sitemap>
> 
> For the time being, I'll assume that we *don't* need this functionality,
> but I'll try and leave it as an option. I really *really* need some
> input from you guys about what you'll be using sitemap based aggregation
> for, and how you envisage it working. The solution will only be as good
> as the input I recieve.

Cool example, I like it! Being able to have sub-pipelines in the
map:part elements might be a bit of an overkill, but the ability to have
transforms inside the part elements might be usefull.  If the generated
page is going to be huge, running small transforms over the content
before agregation may be faster than running multiple transforms over
the agregated content to do the same thing.

What are you current plans for the implementation?

Ross Burton