Integrate Sling with data from another system

[tao wen <ta...@gmail.com>]

Integrate Sling with data from another system



We are working a public website using CQ 5. Some of the data like product
catalogue comes from another system. The requirement asks us to get the data
and embed them as part of the page hosted by CQ. This is a challenging job
to us, as we can not find any example out there doing this. The challenges
we have identified are:



1. How to get/load the data

2. How the data being updated and notified

3. How the data being rendered

4. How to embed them as part of the page



Before we sending out this mail to ask for help, we did some consulting with
sales of CQ, and did some homework like reading source code of sling. Here
are our planned apporaches. We need some advices here about are those
apporaches following the best practices of sling or simply anti-patterns.



1.       How to get/load the data

Some background: we will deploy three CQ instances. One author instance and
two publish instances. The data will be prepared in author instance using
JDBC, and doing some transformation in memory. The result will be persisted
into a xml file and saved in JCR repository. We rely on replication
functionality to publish the xml file to the publish instance. Then in the
publish instance we just need to read things in xml back into memory.



2.       How the data being updated and notified

We will set up a scheduler in author instance to poll the data every day.
The update will be notified as event. If the replication process can
generate event itself, then we will rely on it. Otherwise, we need to set up
new topic and publish event to it when the serialized xml file being
updated. We assume CQ will distribute the event to publish instance so that
the publish instance can know they need to refresh the data in memory using
the updated xml file.



3.       How the data being rendered

As the data is not stored in JCR repository as nodes, but a plain file, so
things like /mobile/handset/NokiaN75 can not be understand by CQ. To make
those nodes available, we write our own resource provider, and map it to
/mobile/handset. We will do URL parsing ourself to get the parameters. The
final resource provided to script engine will contain the necessary data. In
the page, we can reference the handset by reference the current node (a.k.a
the resource provided). The resource provider is similar to tranditional
controller in MVC architecture. So there might be 20+ controllers in a
system, would that be a issue?



4.       How to embed them as part of the page

We leverage the CQ component here. The component will be configured with a
URL, where it can fetch its html. So, the comonent can not render itself,
instead, it relies on another URL to do it (which is backened by a
controller). To do this, we either need to implement some sort of server
side include, or using client side ajax call. The concern of server side
include is how to do that, and how to pass user session. The concern of
using client side ajax call is the user experience might be impacted as the
network could be slow. We were told from the sales of CQ that to be
effective about page caching, we’d better to use ajax call to retrieve the
dynamic content.



Any comments?

Re: Integrate Sling with data from another system

[Rory Douglas <ro...@oracle.com>]

At one point I experimented with a custom ResourceProvider that provided 
REST-ful access to database tables.  The idea works conceptually, but 
requires that you re-create a fair amount of functionality that already 
works out-of-the-box with JCR Node Resources.  I think that's what you 
alluded to with the discussion of URL parsing & 20+ controllers.  
However, I think you could use this idea, and then dispense with 
actually importing the data as XML into the repository & rather just 
access it directly from the source system (if it's a DB, service-bus, 
file-system etc).

In particular, it appears the Sling-side of your application doesn't 
require updating this data, so that would simplify the implementation.  
I eventually abandoned my approach because I didn't have time to 
re-implement all the SlingPostServlet functionality for creation/update 
(which is JCR node specific) for my custom Resource Types.

tao wen wrote:
> Hi,
>
> There are two issues stopped us from importing them as JCR nodes:
>
> 1. the data not owned by us
> It is owned by a backend system. So, it will need to be updated once a day
> at least.
>
> 2. the data has rules
> For example, products have dependencies. When we show the product catalogue,
> we allow user to play with different packaging possibilities. We need to
> apply the domain logic to validate the package user built. So, if we store
> things in JCR nodes, we need to use OCM to map it back to memory and invoke
> domain logic. Also, the relationships of domain entities are very rich, also
> add complexitity to the mappings.
>
> 2009/3/16 Michael Marth <mm...@day.com>
>   

Re: Integrate Sling with data from another system

[tao wen <ta...@gmail.com>]

Hi,

There are two issues stopped us from importing them as JCR nodes:

1. the data not owned by us
It is owned by a backend system. So, it will need to be updated once a day
at least.

2. the data has rules
For example, products have dependencies. When we show the product catalogue,
we allow user to play with different packaging possibilities. We need to
apply the domain logic to validate the package user built. So, if we store
things in JCR nodes, we need to use OCM to map it back to memory and invoke
domain logic. Also, the relationships of domain entities are very rich, also
add complexitity to the mappings.

2009/3/16 Michael Marth <mm...@day.com>

> Hi,
>
> [I believe that large parts of your question are of interest for the Sling
> group, so I'll answer here. For the parts that are related to Communique
> only you might want to head over to
> http://groups.google.com/group/day-communique]
>
> As far as I understand it you try do to the import in a way that is
> possible, but not very JCR-like:
>
> "As the data is not stored in JCR repository as nodes, but a plain file"
>
> I would recommend to actually store each of your products in a JCR node at
> a
> path like /mobile/handset/NokiaN75 or /content/mobile/handset/NokiaN75
> Like that the script resolution is handled by Sling and you will not need
> the "20+ controllers".
>
> The import itself could be done with a little standalone JCR app.
> IIRC Sling can also process POSTed XML files (and produce nodes) so if you
> can control the incoming XML format that might be even simpler (but I would
> personally go the first route)
>
> hth
> Michael
>
> On Sun, Mar 15, 2009 at 7:21 AM, tao wen <ta...@gmail.com> wrote:
>
> > Integrate Sling with data from another system
> >
> >
> >
> > We are working a public website using CQ 5. Some of the data like product
> > catalogue comes from another system. The requirement asks us to get the
> > data
> > and embed them as part of the page hosted by CQ. This is a challenging
> job
> > to us, as we can not find any example out there doing this. The
> challenges
> > we have identified are:
> >
> >
> >
> > 1. How to get/load the data
> >
> > 2. How the data being updated and notified
> >
> > 3. How the data being rendered
> >
> > 4. How to embed them as part of the page
> >
> >
> >
> > Before we sending out this mail to ask for help, we did some consulting
> > with
> > sales of CQ, and did some homework like reading source code of sling.
> Here
> > are our planned apporaches. We need some advices here about are those
> > apporaches following the best practices of sling or simply anti-patterns.
> >
> >
> >
> > 1.       How to get/load the data
> >
> > Some background: we will deploy three CQ instances. One author instance
> and
> > two publish instances. The data will be prepared in author instance using
> > JDBC, and doing some transformation in memory. The result will be
> persisted
> > into a xml file and saved in JCR repository. We rely on replication
> > functionality to publish the xml file to the publish instance. Then in
> the
> > publish instance we just need to read things in xml back into memory.
> >
> >
> >
> > 2.       How the data being updated and notified
> >
> > We will set up a scheduler in author instance to poll the data every day.
> > The update will be notified as event. If the replication process can
> > generate event itself, then we will rely on it. Otherwise, we need to set
> > up
> > new topic and publish event to it when the serialized xml file being
> > updated. We assume CQ will distribute the event to publish instance so
> that
> > the publish instance can know they need to refresh the data in memory
> using
> > the updated xml file.
> >
> >
> >
> > 3.       How the data being rendered
> >
> > As the data is not stored in JCR repository as nodes, but a plain file,
> so
> > things like /mobile/handset/NokiaN75 can not be understand by CQ. To make
> > those nodes available, we write our own resource provider, and map it to
> > /mobile/handset. We will do URL parsing ourself to get the parameters.
> The
> > final resource provided to script engine will contain the necessary data.
> > In
> > the page, we can reference the handset by reference the current node
> (a.k.a
> > the resource provided). The resource provider is similar to tranditional
> > controller in MVC architecture. So there might be 20+ controllers in a
> > system, would that be a issue?
> >
> >
> >
> > 4.       How to embed them as part of the page
> >
> > We leverage the CQ component here. The component will be configured with
> a
> > URL, where it can fetch its html. So, the comonent can not render itself,
> > instead, it relies on another URL to do it (which is backened by a
> > controller). To do this, we either need to implement some sort of server
> > side include, or using client side ajax call. The concern of server side
> > include is how to do that, and how to pass user session. The concern of
> > using client side ajax call is the user experience might be impacted as
> the
> > network could be slow. We were told from the sales of CQ that to be
> > effective about page caching, we’d better to use ajax call to retrieve
> the
> > dynamic content.
> >
> >
> >
> > Any comments?
> >
>
>
>
> --
> Michael Marth | http://dev.day.com/
>

Re: Integrate Sling with data from another system

[Michael Marth <mm...@day.com>]

Hi,

[I believe that large parts of your question are of interest for the Sling
group, so I'll answer here. For the parts that are related to Communique
only you might want to head over to
http://groups.google.com/group/day-communique]

As far as I understand it you try do to the import in a way that is
possible, but not very JCR-like:

"As the data is not stored in JCR repository as nodes, but a plain file"

I would recommend to actually store each of your products in a JCR node at a
path like /mobile/handset/NokiaN75 or /content/mobile/handset/NokiaN75
Like that the script resolution is handled by Sling and you will not need
the "20+ controllers".

The import itself could be done with a little standalone JCR app.
IIRC Sling can also process POSTed XML files (and produce nodes) so if you
can control the incoming XML format that might be even simpler (but I would
personally go the first route)

hth
Michael

On Sun, Mar 15, 2009 at 7:21 AM, tao wen <ta...@gmail.com> wrote:

> Integrate Sling with data from another system
>
>
>
> We are working a public website using CQ 5. Some of the data like product
> catalogue comes from another system. The requirement asks us to get the
> data
> and embed them as part of the page hosted by CQ. This is a challenging job
> to us, as we can not find any example out there doing this. The challenges
> we have identified are:
>
>
>
> 1. How to get/load the data
>
> 2. How the data being updated and notified
>
> 3. How the data being rendered
>
> 4. How to embed them as part of the page
>
>
>
> Before we sending out this mail to ask for help, we did some consulting
> with
> sales of CQ, and did some homework like reading source code of sling. Here
> are our planned apporaches. We need some advices here about are those
> apporaches following the best practices of sling or simply anti-patterns.
>
>
>
> 1.       How to get/load the data
>
> Some background: we will deploy three CQ instances. One author instance and
> two publish instances. The data will be prepared in author instance using
> JDBC, and doing some transformation in memory. The result will be persisted
> into a xml file and saved in JCR repository. We rely on replication
> functionality to publish the xml file to the publish instance. Then in the
> publish instance we just need to read things in xml back into memory.
>
>
>
> 2.       How the data being updated and notified
>
> We will set up a scheduler in author instance to poll the data every day.
> The update will be notified as event. If the replication process can
> generate event itself, then we will rely on it. Otherwise, we need to set
> up
> new topic and publish event to it when the serialized xml file being
> updated. We assume CQ will distribute the event to publish instance so that
> the publish instance can know they need to refresh the data in memory using
> the updated xml file.
>
>
>
> 3.       How the data being rendered
>
> As the data is not stored in JCR repository as nodes, but a plain file, so
> things like /mobile/handset/NokiaN75 can not be understand by CQ. To make
> those nodes available, we write our own resource provider, and map it to
> /mobile/handset. We will do URL parsing ourself to get the parameters. The
> final resource provided to script engine will contain the necessary data.
> In
> the page, we can reference the handset by reference the current node (a.k.a
> the resource provided). The resource provider is similar to tranditional
> controller in MVC architecture. So there might be 20+ controllers in a
> system, would that be a issue?
>
>
>
> 4.       How to embed them as part of the page
>
> We leverage the CQ component here. The component will be configured with a
> URL, where it can fetch its html. So, the comonent can not render itself,
> instead, it relies on another URL to do it (which is backened by a
> controller). To do this, we either need to implement some sort of server
> side include, or using client side ajax call. The concern of server side
> include is how to do that, and how to pass user session. The concern of
> using client side ajax call is the user experience might be impacted as the
> network could be slow. We were told from the sales of CQ that to be
> effective about page caching, we’d better to use ajax call to retrieve the
> dynamic content.
>
>
>
> Any comments?
>



-- 
Michael Marth | http://dev.day.com/