You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by Edgar Poce <ed...@gmail.com> on 2005/03/02 18:40:12 UTC
Chained Persistence and Filesystem
Hi to all
I need to store data with many persistence managers, ORMPersistence for
some custom node types, XMLPersistence in mem for temp data and
XMlPersistence on disk for the rest.
I'm planning to make a ChainedPersistence and a ChainedFileSystem
manager, both implementing the chain of responsibility pattern.
The general idea is to wrap the PersistenceManager in a persistence
handler which delegates the call if a certain condition is satisfied.
Initially, I would need a UriPersistenceHandler and a
NodeTypePersistenceHandler.
Is there any plan to include this feature in the short term? Is it a
good approach?
Any feedback is welcome.
thanks for your support
edgar
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi marcel
Marcel Reutegger wrote:
> Hi Edgar,
>
> Edgar Poce wrote:
>
>> I'm planning to make a ChainedPersistence and a ChainedFileSystem
>> manager, both implementing the chain of responsibility pattern.
>
>
> that's an interesting idea. I was recently thinking about a stacked
> persistence manager (PM) that uses a persistent PM and has an in memory
> PM on top of it. the use case behind that is to run jackrabbit off a
> read-only filesystem but still being able to perform write operations.
>
>> The general idea is to wrap the PersistenceManager in a persistence
>> handler which delegates the call if a certain condition is satisfied.
>> Initially, I would need a UriPersistenceHandler and a
>> NodeTypePersistenceHandler.
>>
>> Is there any plan to include this feature in the short term? Is it a
>> good approach?
>
>
> We didn't discuss such a feature specifically, but I think it would make
> an interesting contribution, that we might move to the core sometime.
>
I meant to ask if a similar work was in progress just to don't start
something that will be available in a short term. It was not my
intention to ask whether my implementation would be eligible. I read my
mail again and I realized I wasn't clear.
> cheers
> marcel
>
regards
edgar
Re: Chained Persistence and Filesystem
Posted by Marcel Reutegger <ma...@gmx.net>.
Hi Edgar,
Edgar Poce wrote:
> I'm planning to make a ChainedPersistence and a ChainedFileSystem
> manager, both implementing the chain of responsibility pattern.
that's an interesting idea. I was recently thinking about a stacked
persistence manager (PM) that uses a persistent PM and has an in memory
PM on top of it. the use case behind that is to run jackrabbit off a
read-only filesystem but still being able to perform write operations.
> The general idea is to wrap the PersistenceManager in a persistence
> handler which delegates the call if a certain condition is satisfied.
> Initially, I would need a UriPersistenceHandler and a
> NodeTypePersistenceHandler.
>
> Is there any plan to include this feature in the short term? Is it a
> good approach?
We didn't discuss such a feature specifically, but I think it would make
an interesting contribution, that we might move to the core sometime.
cheers
marcel
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi
If anyone wants to give it a try, I uploaded a the
ChainedPersistenceManager to jira.
http://issues.apache.org/jira/browse/JCR-65
Any feedback is welcome.
regards
edgar
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
hi david
I'm sorry I wasn't clear. I'm bussy right now but I'll send you a
descriptive doc to your personal address in couple of days. It's out of
topic and I guess many have no interest.
David Nuescheler wrote:
> hi edgar,
>
> unfortunately, your diagram didnt make it all the way
> through to me. did anyone else get it? do you
> have an uri?
Sorry, but no uri yet.
>
> regards,
> david
>
regards
Edgar
Re: Chained Persistence and Filesystem
Posted by David Nuescheler <da...@gmail.com>.
hi edgar,
unfortunately, your diagram didnt make it all the way
through to me. did anyone else get it? do you
have an uri?
regards,
david
On Sun, 06 Mar 2005 14:42:55 -0300, Edgar Poce <ed...@gmail.com> wrote:
> Hi david
>
> David Nuescheler wrote:
> > since i am very interested in making performance
> > improvements to jackrabbits query engine where
> > necessary, i would really be interested in the nature
> > and frequency of your queries, so we have something
> > to base optimization on. do you think you can
> > share details?
>
> I attach a small diagram (8.5 k).
>
> Now the only source provider implementation I've got is using jcr to
> cache the query results. It uses JDBC to execute the queries but I'm
> open to jcr if it has a minimal set of tools for running descriptive
> statistics.
>
> I'm also using JCR in the web app.
>
> regards
> edgar
>
>
--
----------------------------------------------------------------------
standardize your content-repository !
http://www.jcp.org/en/jsr/detail?id=170
---------------------------------------< david.nuescheler@day.com >---
This message is a private communication. If you are not the intended
recipient, please do not read, copy, or use it, and do not disclose it
to others. Please notify the sender of the delivery error by replying
to this message, and then delete it from your system. Thank you.
The sender does not assume any liability for timely, trouble free,
complete, virus free, secure, error free or uninterrupted arrival of
this e-mail. For verification please request a hard copy version.
mailto:david.nuescheler@day.com
http://www.day.com
David Nuescheler
Chief Technology Officer
Day Software AG
Barfuesserplatz 6 / Postfach
4001 Basel
Switzerland
T 41 61 226 98 98
F 41 61 226 98 97
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi david
David Nuescheler wrote:
> since i am very interested in making performance
> improvements to jackrabbits query engine where
> necessary, i would really be interested in the nature
> and frequency of your queries, so we have something
> to base optimization on. do you think you can
> share details?
I attach a small diagram (8.5 k).
Now the only source provider implementation I’ve got is using jcr to
cache the query results. It uses JDBC to execute the queries but I’m
open to jcr if it has a minimal set of tools for running descriptive
statistics.
I’m also using JCR in the web app.
regards
edgar
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi david
David Nuescheler wrote:
> hi edgar,
>
>
>>I'm far (really far) from being a db expert, and I still don't get the
>>big picture of jackrabbit internals, but I think jackrabbit is unable,
>>and probably will allways be, to perform aggregated queries as fast as
>>any of the os or commercial dbs.
>
> why would that be? the querymanager could be extended
> with an additional query language that does a direct
> pass-through (i would not necessarily suggest that. in
> agreement with jukkas statement about trying to avoid
> relying on the structure of the persistance layer).
> also, i would argue that depending on the nature of the
> query jackrabbit may have a more adequate index and
> therefore may easily outperform a classical rdbms.
>
I didn't see any of these options. It's a little more clear now :). thanks.
I took a look to the jcr searching chapter and I didn't find any
reference to aggregated functions. A minimal set of functions (avg,
mean, std, count, min and max among others) would be enough for
descriptive statistics. And a pluggable mechanism for user defined
functions would be very useful for more complex calculations. Something
like udf in mysql. See http://dev.mysql.com/doc/mysql/en/adding-udf.html.
Is this in the scope of jcr?. and, how do I achieve this functionality
through the jcr api in its current state?.
> you mentioned reporting, in reporting in general
> performance usually is not so much an issue, since
> in many cases reporting is not done very frequently.
>
I agree, even in some cases where it's done frecuently. In my project
I'm using a query handler which is responsible of executing queries.
Depending on the estimated time it uses synchronous or asynchronous
strategies, with a single or multiple threads. Query results are
persisted (here I use jackrabbit too) and will be available for sync
reporting the next time.
> since i am very interested in making performance
> improvements to jackrabbits query engine where
> necessary, i would really be interested in the nature
> and frequency of your queries, so we have something
> to base optimization on. do you think you can
> share details?
>
There are two kind of queries. One for microdata processing (mostly
async reporting), that's where I need aggregated functions, and another
for sync reporting. I still don't know the frequency of the queries
because it's not yet in production and will not be until a few months.
Aggregated queries will include cases ranging from 0 to the entire
population size by the number of waves. Now I'm handling only one
household survey with 100k per wave, aproximatly 6 million cases total.
I made a pluggable mechanism for adding source providers and I plan to
add many more sources, censuses and mainly household surveys.
> jsr-170 specifies persistent queries, which allows
> for substancial optimizations. do you think an
> extension of the persisent queries could help you
> with respect to performance?
>
I still don't know, but I will take a look to the jcr doc and the
jackrabbit implementation and see if it fits my needs. Thanks for the hint.
> regards,
> david
>
Thank you very much
edgar
Re: Chained Persistence and Filesystem
Posted by David Nuescheler <da...@gmail.com>.
hi edgar,
> I'm far (really far) from being a db expert, and I still don't get the
> big picture of jackrabbit internals, but I think jackrabbit is unable,
> and probably will allways be, to perform aggregated queries as fast as
> any of the os or commercial dbs.
why would that be? the querymanager could be extended
with an additional query language that does a direct
pass-through (i would not necessarily suggest that. in
agreement with jukkas statement about trying to avoid
relying on the structure of the persistance layer).
also, i would argue that depending on the nature of the
query jackrabbit may have a more adequate index and
therefore may easily outperform a classical rdbms.
you mentioned reporting, in reporting in general
performance usually is not so much an issue, since
in many cases reporting is not done very frequently.
since i am very interested in making performance
improvements to jackrabbits query engine where
necessary, i would really be interested in the nature
and frequency of your queries, so we have something
to base optimization on. do you think you can
share details?
jsr-170 specifies persistent queries, which allows
for substancial optimizations. do you think an
extension of the persisent queries could help you
with respect to performance?
regards,
david
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi
Jukka Zitting wrote:
> Hi,
>
> Edgard Poce wrote:
>
>>I comment my use case. I'm working with a big amount of data, censuses
>>and other household surveys. I'd like to store each survey with
>>jackrabbit, but I'd like/need jackrabbit could share the data with any
>>statistical processing engine because I don't feel like implementing
>>statistical algorithms. That's why I need a custom orm pm for
>>myapp:mysurvey nodetype. What's the best practice?
>
>
> There are few best practices yet, but in my opinion you'd better focus
> on doing the integration on top of the JCR API, not below it. Is it
> possible to implement a JCR storage adapter for the statistical
> processing engine you are using? Or alternatively implement an export
> feature that extracts the required data from a JCR content repository.
>
i think that exporting a 1gb database database is not an option. Besides
I should write the code to perform the conversion from jcr api to a
readable db format, I guess it's not so hard but there's no need to do it.
> You can think of the Jackrabbit persistence manager layer as the storage
> layer of an RDBMS implementation. The JCR API is to Jackrabbit what JDBC
> is to an RDBMS. You can use a database to store your data and integrate
> it with your applications using the public API, but you certainly
> wouldn't want to try making your externally stored data visible inside
> the database by writing a custom storage layer implementation... ;-)
>
I don't fully understand what you said. :(
I want to store, edit, browse, data through the jcr api, and make the
statistical processing directly from the persistence storage. It would
be just a read only task, so it shouldn't affect the jackrabbit normal
behaviour.
I'm far (really far) from being a db expert, and I still don't get the
big picture of jackrabbit internals, but I think jackrabbit is unable,
and probably will allways be, to perform aggregated queries as fast as
any of the os or commercial dbs.
> BR,
>
> Jukka Zitting
>
>
>
regards
edgar
Re: Chained Persistence and Filesystem
Posted by Jukka Zitting <jz...@yukatan.fi>.
Hi,
Edgard Poce wrote:
> I comment my use case. I'm working with a big amount of data, censuses
> and other household surveys. I'd like to store each survey with
> jackrabbit, but I'd like/need jackrabbit could share the data with any
> statistical processing engine because I don't feel like implementing
> statistical algorithms. That's why I need a custom orm pm for
> myapp:mysurvey nodetype. What's the best practice?
There are few best practices yet, but in my opinion you'd better focus
on doing the integration on top of the JCR API, not below it. Is it
possible to implement a JCR storage adapter for the statistical
processing engine you are using? Or alternatively implement an export
feature that extracts the required data from a JCR content repository.
You can think of the Jackrabbit persistence manager layer as the storage
layer of an RDBMS implementation. The JCR API is to Jackrabbit what JDBC
is to an RDBMS. You can use a database to store your data and integrate
it with your applications using the public API, but you certainly
wouldn't want to try making your externally stored data visible inside
the database by writing a custom storage layer implementation... ;-)
BR,
Jukka Zitting
Re: Chained Persistence and Filesystem
Posted by Edgar Poce <ed...@gmail.com>.
Hi
Stefan Guggisberg wrote:
> hi edgar,
>
> On Wed, 02 Mar 2005 14:40:12 -0300, Edgar Poce <ed...@gmail.com> wrote:
>
>>Hi to all
>>
>>I need to store data with many persistence managers, ORMPersistence for
>>some custom node types, XMLPersistence in mem for temp data and
>>XMlPersistence on disk for the rest.
>>
>>I'm planning to make a ChainedPersistence and a ChainedFileSystem
>>manager, both implementing the chain of responsibility pattern.
>>
>>The general idea is to wrap the PersistenceManager in a persistence
>>handler which delegates the call if a certain condition is satisfied.
>>Initially, I would need a UriPersistenceHandler and a
>>NodeTypePersistenceHandler.
>
>
> sounds interesting. do you mind sharing your design with us in a
> few sentences?
>
see
http://www.javaworld.com/javaworld/jw-08-2003/jw-0829-designpatterns.html
figure 2.
Each concrete handler is a PersistenceManager wrapper and delegates the
call only if certain condition is satisfied. Initially I thought of
handlers that performs the evaluation based on the uri
(UriPersistenceHandler), on the node type (NodeTypePersistenceHandler)
and a DefaultPersistenceHandler which always delegates it.
>
>>Is there any plan to include this feature in the short term? Is it a
>>good approach?
>
>
> i am bit worried about doing fancy stuff in PersistenceManagers
> in general. let me explain...
>
> the PersistenceManager interface was never intended as being
> a general SPI that you could implement in order to integrate
> external datasources with proprietary formats (e.g. a customers
> database).
>
> the reason why we abstracted the PersistenceManager interface
> was to leave room for future performance optimizations that
> would not affect the rest of the implementation (e.g. by storing
> the raw data in a b-tree based database instead of individual file).
>
> the PersistenceManager sits at the very bottom layer in
> jackrabbits (not yet existing... :( system architecture diagram.
> reliability, integrity and performance of the PersistenceManager
> are *crucial* to the overall stability & performance of the repository.
> if e.g. the data that a PersistenceManager is based upon is
> allowed to change through external means the integrity
> of the repository would be at risk (think of referential integrity/
> node references e.g.).
>
> therefore, imo, a PersistenceManager should not be 'intelligent',
> it should not 'interpret' the data. the only thing it should care about
> is to efficiently, consistently and reliably store and read the data
> encapsulated in the passed NodeState & PropertyState objects.
>
> though it might be feasible to write a custom persistence manager
> to represent existing legacy data in a level-1 (read-only) repository,
> i don't think the same is possible for a level-2 repository and i
> certainly would not recommend it.
>
I'm really confused now. Why level 1 and 2 are mutually excluded? I
thought a level 2 repo should have at least all the features of the
level 1.
If, as you said, I shouldn't chain persistence managers, if I had a few
legacy systems I would need a jcr instance for each one?
And if I had a level 2 repo, it would be *only* a level 2 and I
shouldn't integrate any other source?
I comment my use case. I'm working with a big amount of data, censuses
and other household surveys. I'd like to store each survey with
jackrabbit, but I'd like/need jackrabbit could share the data with any
statistical processing engine because I don't feel like implementing
statistical algorithms. That's why I need a custom orm pm for
myapp:mysurvey nodetype. What's the best practice?
> i hope i didn't scare you off, i just wanted to make sure that you are
> fully aware of all potential risk & implications when writing
> custom persistence managers.
>
> cheers
> stefan
>
>
>>Any feedback is welcome.
>>
>>thanks for your support
>>edgar
>>
>>
>
>
thanks
edgar
Re: Chained Persistence and Filesystem
Posted by Stefan Guggisberg <st...@gmail.com>.
hi edgar,
On Wed, 02 Mar 2005 14:40:12 -0300, Edgar Poce <ed...@gmail.com> wrote:
> Hi to all
>
> I need to store data with many persistence managers, ORMPersistence for
> some custom node types, XMLPersistence in mem for temp data and
> XMlPersistence on disk for the rest.
>
> I'm planning to make a ChainedPersistence and a ChainedFileSystem
> manager, both implementing the chain of responsibility pattern.
>
> The general idea is to wrap the PersistenceManager in a persistence
> handler which delegates the call if a certain condition is satisfied.
> Initially, I would need a UriPersistenceHandler and a
> NodeTypePersistenceHandler.
sounds interesting. do you mind sharing your design with us in a
few sentences?
>
> Is there any plan to include this feature in the short term? Is it a
> good approach?
i am bit worried about doing fancy stuff in PersistenceManagers
in general. let me explain...
the PersistenceManager interface was never intended as being
a general SPI that you could implement in order to integrate
external datasources with proprietary formats (e.g. a customers
database).
the reason why we abstracted the PersistenceManager interface
was to leave room for future performance optimizations that
would not affect the rest of the implementation (e.g. by storing
the raw data in a b-tree based database instead of individual file).
the PersistenceManager sits at the very bottom layer in
jackrabbits (not yet existing... :( system architecture diagram.
reliability, integrity and performance of the PersistenceManager
are *crucial* to the overall stability & performance of the repository.
if e.g. the data that a PersistenceManager is based upon is
allowed to change through external means the integrity
of the repository would be at risk (think of referential integrity/
node references e.g.).
therefore, imo, a PersistenceManager should not be 'intelligent',
it should not 'interpret' the data. the only thing it should care about
is to efficiently, consistently and reliably store and read the data
encapsulated in the passed NodeState & PropertyState objects.
though it might be feasible to write a custom persistence manager
to represent existing legacy data in a level-1 (read-only) repository,
i don't think the same is possible for a level-2 repository and i
certainly would not recommend it.
i hope i didn't scare you off, i just wanted to make sure that you are
fully aware of all potential risk & implications when writing
custom persistence managers.
cheers
stefan
>
> Any feedback is welcome.
>
> thanks for your support
> edgar
>
>