You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by Edgar Poce <ed...@gmail.com> on 2005/03/02 18:40:12 UTC

Chained Persistence and Filesystem

Hi to all

I need to store data with many persistence managers, ORMPersistence for 
some custom node types, XMLPersistence in mem for temp data and 
XMlPersistence on disk for the rest.

I'm planning to make a ChainedPersistence and a ChainedFileSystem 
manager, both implementing the chain of responsibility pattern.

The general idea is to wrap the PersistenceManager in a persistence 
handler which delegates the call if a certain condition is satisfied. 
Initially, I would need a UriPersistenceHandler and a 
NodeTypePersistenceHandler.

Is there any plan to include this feature in the short term? Is it a 
good approach?

Any feedback is welcome.

thanks for your support
edgar


Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi marcel

Marcel Reutegger wrote:
> Hi Edgar,
> 
> Edgar Poce wrote:
> 
>> I'm planning to make a ChainedPersistence and a ChainedFileSystem 
>> manager, both implementing the chain of responsibility pattern.
> 
> 
> that's an interesting idea. I was recently thinking about a stacked 
> persistence manager (PM) that uses a persistent PM and has an in memory 
> PM on top of it. the use case behind that is to run jackrabbit off a 
> read-only filesystem but still being able to perform write operations.
> 
>> The general idea is to wrap the PersistenceManager in a persistence 
>> handler which delegates the call if a certain condition is satisfied. 
>> Initially, I would need a UriPersistenceHandler and a 
>> NodeTypePersistenceHandler.
>>
>> Is there any plan to include this feature in the short term? Is it a 
>> good approach?
> 
> 
> We didn't discuss such a feature specifically, but I think it would make 
> an interesting contribution, that we might move to the core sometime.
> 

I meant to ask if a similar work was in progress just to don't start 
something that will be available in a short term. It was not my 
intention to ask whether my implementation would be eligible. I read my 
mail again and I realized I wasn't clear.

> cheers
>  marcel
> 

regards
edgar

Re: Chained Persistence and Filesystem

Posted by Marcel Reutegger <ma...@gmx.net>.
Hi Edgar,

Edgar Poce wrote:
> I'm planning to make a ChainedPersistence and a ChainedFileSystem 
> manager, both implementing the chain of responsibility pattern.

that's an interesting idea. I was recently thinking about a stacked 
persistence manager (PM) that uses a persistent PM and has an in memory 
PM on top of it. the use case behind that is to run jackrabbit off a 
read-only filesystem but still being able to perform write operations.

> The general idea is to wrap the PersistenceManager in a persistence 
> handler which delegates the call if a certain condition is satisfied. 
> Initially, I would need a UriPersistenceHandler and a 
> NodeTypePersistenceHandler.
> 
> Is there any plan to include this feature in the short term? Is it a 
> good approach?

We didn't discuss such a feature specifically, but I think it would make 
an interesting contribution, that we might move to the core sometime.

cheers
  marcel

Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi

  If anyone wants to give it a try, I uploaded a the 
ChainedPersistenceManager to jira. 
http://issues.apache.org/jira/browse/JCR-65
Any feedback is welcome.

regards
edgar

Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
hi david

  I'm sorry I wasn't clear. I'm bussy right now but I'll send you a 
descriptive doc to your personal address in couple of days. It's out of 
topic and I guess many have no interest.

David Nuescheler wrote:
> hi edgar,
> 
> unfortunately, your diagram didnt make it all the way 
> through to me. did anyone else get it? do you
> have an uri?

Sorry, but no uri yet.

> 
> regards,
> david
> 

regards
Edgar


Re: Chained Persistence and Filesystem

Posted by David Nuescheler <da...@gmail.com>.
hi edgar,

unfortunately, your diagram didnt make it all the way 
through to me. did anyone else get it? do you
have an uri?

regards,
david


On Sun, 06 Mar 2005 14:42:55 -0300, Edgar Poce <ed...@gmail.com> wrote:
> Hi david
> 
> David Nuescheler wrote:
> > since i am very interested in making performance
> > improvements to jackrabbits query engine where
> > necessary, i would really be interested in the nature
> > and frequency of your queries, so we have something
> > to base optimization on. do you think you can
> > share details?
> 
> I attach a small diagram (8.5 k).
> 
> Now the only source provider implementation I've got is using jcr to
> cache the query results. It uses JDBC to execute the queries but I'm
> open to jcr if it has a minimal set of tools for running descriptive
> statistics.
> 
> I'm also using JCR in the web app.
> 
> regards
> edgar
> 
> 


-- 
----------------------------------------------------------------------
standardize your content-repository !
                               http://www.jcp.org/en/jsr/detail?id=170
---------------------------------------< david.nuescheler@day.com >---

This message is a private communication. If you are not the intended
recipient, please do not read, copy, or use it, and do not disclose it
to others. Please notify the sender of the delivery error by replying
to this message, and then delete it from your system. Thank you.

The sender does not assume any liability for timely, trouble free,
complete, virus free, secure, error free or uninterrupted arrival of
this e-mail. For verification please request a hard copy version.


mailto:david.nuescheler@day.com
http://www.day.com

David Nuescheler
Chief Technology Officer
Day Software AG
Barfuesserplatz 6 / Postfach
4001 Basel
Switzerland

T  41 61 226 98 98
F  41 61 226 98 97

Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi david

David Nuescheler wrote:
> since i am very interested in making performance
> improvements to jackrabbits query engine where
> necessary, i would really be interested in the nature
> and frequency of your queries, so we have something
> to base optimization on. do you think you can 
> share details?

I attach a small diagram (8.5 k).

Now the only source provider implementation I’ve got is using jcr to 
cache the query results. It uses JDBC to execute the queries but I’m 
open to jcr if it has a minimal set of tools for running descriptive 
statistics.

I’m also using JCR in the web app.

regards
edgar

Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi david

David Nuescheler wrote:
> hi edgar,
> 
> 
>>I'm far (really far) from being a db expert, and I still don't get the
>>big picture of jackrabbit internals, but I think jackrabbit is unable,
>>and probably will allways be, to perform aggregated queries as fast as
>>any of the os or commercial dbs.
> 
> why would that be? the querymanager could be extended
> with an additional query language that does a direct 
> pass-through (i would not necessarily suggest that. in 
> agreement with jukkas statement about trying to avoid 
> relying on the structure of the persistance layer).
> also, i would argue that depending on the nature of the 
> query jackrabbit may have a more adequate index and 
> therefore may easily outperform a classical rdbms.
> 
I didn't see any of these options. It's a little more clear now :). thanks.
I took a look to the jcr searching chapter and I didn't find any 
reference to aggregated functions. A minimal set of functions (avg, 
mean, std, count, min and max among others) would be enough for 
descriptive statistics. And a pluggable mechanism for user defined 
functions would be very useful for more complex calculations. Something 
like udf in mysql. See http://dev.mysql.com/doc/mysql/en/adding-udf.html.
Is this in the scope of jcr?. and, how do I achieve this functionality 
through the jcr api in its current state?.

> you mentioned reporting, in reporting in general 
> performance usually is not so much an issue, since
> in many cases reporting is not done very frequently.
> 
I agree, even in some cases where it's done frecuently. In my project 
I'm using a query handler which is responsible of executing queries. 
Depending on the estimated time it uses synchronous or asynchronous 
strategies, with a single or multiple threads. Query results are 
persisted (here I use jackrabbit too) and will be available for sync 
reporting the next time.

> since i am very interested in making performance
> improvements to jackrabbits query engine where
> necessary, i would really be interested in the nature
> and frequency of your queries, so we have something
> to base optimization on. do you think you can 
> share details?
> 
There are two kind of queries. One for microdata processing (mostly 
async reporting), that's where I need aggregated functions, and another 
for sync reporting. I still don't know the frequency of the queries 
because it's not yet in production and will not be until a few months.
Aggregated queries will include cases ranging from 0 to the entire 
population size by the number of waves. Now I'm handling only one 
household survey with 100k per wave, aproximatly 6 million cases total.
I made a pluggable mechanism for adding source providers and I plan to 
add many more sources, censuses and mainly household surveys.

> jsr-170 specifies persistent queries, which allows
> for substancial optimizations. do you think an 
> extension of the persisent queries could help you
> with respect to performance?
> 
I still don't know, but I will take a look to the jcr doc and the 
jackrabbit implementation and see if it fits my needs. Thanks for the hint.

> regards,
> david
> 

Thank you very much
edgar

Re: Chained Persistence and Filesystem

Posted by David Nuescheler <da...@gmail.com>.
hi edgar,

> I'm far (really far) from being a db expert, and I still don't get the
> big picture of jackrabbit internals, but I think jackrabbit is unable,
> and probably will allways be, to perform aggregated queries as fast as
> any of the os or commercial dbs.
why would that be? the querymanager could be extended
with an additional query language that does a direct 
pass-through (i would not necessarily suggest that. in 
agreement with jukkas statement about trying to avoid 
relying on the structure of the persistance layer).
also, i would argue that depending on the nature of the 
query jackrabbit may have a more adequate index and 
therefore may easily outperform a classical rdbms.

you mentioned reporting, in reporting in general 
performance usually is not so much an issue, since
in many cases reporting is not done very frequently.

since i am very interested in making performance
improvements to jackrabbits query engine where
necessary, i would really be interested in the nature
and frequency of your queries, so we have something
to base optimization on. do you think you can 
share details?

jsr-170 specifies persistent queries, which allows
for substancial optimizations. do you think an 
extension of the persisent queries could help you
with respect to performance?

regards,
david

Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi

Jukka Zitting wrote:
> Hi,
> 
> Edgard Poce wrote:
> 
>>I comment my use case. I'm working with a big amount of data, censuses 
>>and other household surveys. I'd like to store each survey with 
>>jackrabbit, but I'd like/need jackrabbit could share the data with any 
>>statistical processing engine because I don't feel like implementing 
>>statistical algorithms. That's why I need a custom orm pm for 
>>myapp:mysurvey nodetype. What's the best practice?
> 
> 
> There are few best practices yet, but in my opinion you'd better focus
> on doing the integration on top of the JCR API, not below it. Is it
> possible to implement a JCR storage adapter for the statistical
> processing engine you are using? Or alternatively implement an export
> feature that extracts the required data from a JCR content repository.
> 
i think that exporting a 1gb database database is not an option. Besides 
I should write the code to perform the conversion from jcr api to a 
readable db format, I guess it's not so hard but there's no need to do it.

> You can think of the Jackrabbit persistence manager layer as the storage
> layer of an RDBMS implementation. The JCR API is to Jackrabbit what JDBC
> is to an RDBMS. You can use a database to store your data and integrate
> it with your applications using the public API, but you certainly
> wouldn't want to try making your externally stored data visible inside
> the database by writing a custom storage layer implementation... ;-)
> 
I don't fully understand what you said. :(
I want to store, edit, browse, data through the jcr api, and make the 
statistical processing directly from the persistence storage. It would 
be just a read only task, so it shouldn't affect the jackrabbit normal 
behaviour.

I'm far (really far) from being a db expert, and I still don't get the 
big picture of jackrabbit internals, but I think jackrabbit is unable, 
and probably will allways be, to perform aggregated queries as fast as 
any of the os or commercial dbs.

> BR,
> 
> Jukka Zitting
> 
> 
> 
regards
edgar

Re: Chained Persistence and Filesystem

Posted by Jukka Zitting <jz...@yukatan.fi>.
Hi,

Edgard Poce wrote:
> I comment my use case. I'm working with a big amount of data, censuses 
> and other household surveys. I'd like to store each survey with 
> jackrabbit, but I'd like/need jackrabbit could share the data with any 
> statistical processing engine because I don't feel like implementing 
> statistical algorithms. That's why I need a custom orm pm for 
> myapp:mysurvey nodetype. What's the best practice?

There are few best practices yet, but in my opinion you'd better focus
on doing the integration on top of the JCR API, not below it. Is it
possible to implement a JCR storage adapter for the statistical
processing engine you are using? Or alternatively implement an export
feature that extracts the required data from a JCR content repository.

You can think of the Jackrabbit persistence manager layer as the storage
layer of an RDBMS implementation. The JCR API is to Jackrabbit what JDBC
is to an RDBMS. You can use a database to store your data and integrate
it with your applications using the public API, but you certainly
wouldn't want to try making your externally stored data visible inside
the database by writing a custom storage layer implementation... ;-)

BR,

Jukka Zitting



Re: Chained Persistence and Filesystem

Posted by Edgar Poce <ed...@gmail.com>.
Hi

Stefan Guggisberg wrote:
> hi edgar,
> 
> On Wed, 02 Mar 2005 14:40:12 -0300, Edgar Poce <ed...@gmail.com> wrote:
> 
>>Hi to all
>>
>>I need to store data with many persistence managers, ORMPersistence for
>>some custom node types, XMLPersistence in mem for temp data and
>>XMlPersistence on disk for the rest.
>>
>>I'm planning to make a ChainedPersistence and a ChainedFileSystem
>>manager, both implementing the chain of responsibility pattern.
>>
>>The general idea is to wrap the PersistenceManager in a persistence
>>handler which delegates the call if a certain condition is satisfied.
>>Initially, I would need a UriPersistenceHandler and a
>>NodeTypePersistenceHandler.
> 
> 
> sounds interesting. do you mind sharing your design with us in a 
> few sentences?
> 
see 
http://www.javaworld.com/javaworld/jw-08-2003/jw-0829-designpatterns.html 
figure 2.
Each concrete handler is a PersistenceManager wrapper and delegates the 
call only if certain condition is satisfied. Initially I thought of 
handlers that performs the evaluation based on the uri 
(UriPersistenceHandler), on the node type (NodeTypePersistenceHandler) 
and a DefaultPersistenceHandler which always delegates it.

> 
>>Is there any plan to include this feature in the short term? Is it a
>>good approach?
> 
> 
> i am bit worried about doing fancy stuff in PersistenceManagers 
> in general. let me explain...
> 
> the PersistenceManager interface was never intended as being
> a general SPI that you could implement in order to integrate 
> external datasources with proprietary formats (e.g. a customers 
> database).
> 
> the reason why we abstracted the PersistenceManager interface
> was to leave room for future performance optimizations that 
> would not affect the rest of the implementation (e.g. by storing 
> the raw data in a b-tree based database instead of individual file).
> 
> the PersistenceManager sits at the very bottom layer in
> jackrabbits (not yet existing... :(  system architecture diagram.
> reliability, integrity and performance of the PersistenceManager
> are *crucial* to the overall stability & performance of the repository.
> if e.g. the data that a PersistenceManager is based upon is 
> allowed to change through external means the integrity
> of the repository would be at risk (think of referential integrity/
> node references e.g.).
> 
> therefore, imo, a PersistenceManager  should not be 'intelligent', 
> it should not 'interpret' the data. the only thing it should care about 
> is to efficiently, consistently and reliably store and read the data 
> encapsulated in the passed NodeState & PropertyState objects.
> 
> though it might be feasible to write a custom persistence manager
> to represent existing legacy data in a level-1 (read-only) repository,
> i don't think the same is possible for a level-2 repository and i 
> certainly would not recommend it.
> 
I'm really confused now. Why level 1 and 2 are mutually excluded? I 
thought a level 2 repo should have at least all the features of the 
level 1.

If, as you said, I shouldn't chain persistence managers, if I had a few 
legacy systems I would need a jcr instance for each one?

And if I had a level 2 repo, it would be *only* a level 2 and I 
shouldn't integrate any other source?

I comment my use case. I'm working with a big amount of data, censuses 
and other household surveys. I'd like to store each survey with 
jackrabbit, but I'd like/need jackrabbit could share the data with any 
statistical processing engine because I don't feel like implementing 
statistical algorithms. That's why I need a custom orm pm for 
myapp:mysurvey nodetype. What's the best practice?

> i hope i didn't scare you off, i just wanted to make sure that you are
> fully aware of all potential risk & implications when writing 
> custom persistence managers.
> 
> cheers
> stefan
> 
> 
>>Any feedback is welcome.
>>
>>thanks for your support
>>edgar
>>
>>
> 
> 

thanks
edgar

Re: Chained Persistence and Filesystem

Posted by Stefan Guggisberg <st...@gmail.com>.
hi edgar,

On Wed, 02 Mar 2005 14:40:12 -0300, Edgar Poce <ed...@gmail.com> wrote:
> Hi to all
> 
> I need to store data with many persistence managers, ORMPersistence for
> some custom node types, XMLPersistence in mem for temp data and
> XMlPersistence on disk for the rest.
> 
> I'm planning to make a ChainedPersistence and a ChainedFileSystem
> manager, both implementing the chain of responsibility pattern.
> 
> The general idea is to wrap the PersistenceManager in a persistence
> handler which delegates the call if a certain condition is satisfied.
> Initially, I would need a UriPersistenceHandler and a
> NodeTypePersistenceHandler.

sounds interesting. do you mind sharing your design with us in a 
few sentences?

> 
> Is there any plan to include this feature in the short term? Is it a
> good approach?

i am bit worried about doing fancy stuff in PersistenceManagers 
in general. let me explain...

the PersistenceManager interface was never intended as being
a general SPI that you could implement in order to integrate 
external datasources with proprietary formats (e.g. a customers 
database).

the reason why we abstracted the PersistenceManager interface
was to leave room for future performance optimizations that 
would not affect the rest of the implementation (e.g. by storing 
the raw data in a b-tree based database instead of individual file).

the PersistenceManager sits at the very bottom layer in
jackrabbits (not yet existing... :(  system architecture diagram.
reliability, integrity and performance of the PersistenceManager
are *crucial* to the overall stability & performance of the repository.
if e.g. the data that a PersistenceManager is based upon is 
allowed to change through external means the integrity
of the repository would be at risk (think of referential integrity/
node references e.g.).

therefore, imo, a PersistenceManager  should not be 'intelligent', 
it should not 'interpret' the data. the only thing it should care about 
is to efficiently, consistently and reliably store and read the data 
encapsulated in the passed NodeState & PropertyState objects.

though it might be feasible to write a custom persistence manager
to represent existing legacy data in a level-1 (read-only) repository,
i don't think the same is possible for a level-2 repository and i 
certainly would not recommend it.

i hope i didn't scare you off, i just wanted to make sure that you are
fully aware of all potential risk & implications when writing 
custom persistence managers.

cheers
stefan

> 
> Any feedback is welcome.
> 
> thanks for your support
> edgar
> 
>