You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ojb-user@db.apache.org by Bruno CROS <br...@gmail.com> on 2006/02/06 17:56:21 UTC
Auto-retrieve or not ?
About my precedent batch troubles:
In fact, a saw my model loaded from every where with all the actual
auto-retrieve="true", this means, every where !! This means too, that
"back" relations are read too, circling too much. This was the cause of my
OutOfMemoryError.
My model is a big one with a lot of relations, as complex as you can
imagine.
So, i'm asking me about get rid of those auto-retrieve, to get all objects
reads faster (and avoid a clogging). But i read that for ODMG dev,
precognized settings are auto-retrieve="true" auto-update="none"
auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
At start, I saw that setting auto-retrieve to "true" everywhere wasn't
solution, but all transaction and batch processes were working fine ( until
1.0.4. ), with autoretrieve on all n relations (yes!). Chance !! But with a
little doubt, I tell to all the dev team to avoid as possible the read by
iterating collections without any reasons, prefering ReportQuery (single
shot) and direct object queries.
Is that the good way to have a fast and robust application ?
Another thing, does defaultPersistenceBroker always return the tx broker
when tx has begun (in 1.0.4)? That's very important.
I hope that i have not to implement with TransactionExt.getBroker method.
All ours reads are with defaultPersistenceBroker.
Thank you all in advance.
Re: Auto-retrieve or not ?
Posted by Armin Waibel <ar...@apache.org>.
Hi Bruno,
Bruno CROS wrote:
> (to Armin) OK. Really do appreciate all your advices . Thanks again.
>
> Here's my situation, resulting of a migration from 1.0.1 to 1.0.4 (within
> ojb-blank) :
>
> - I tried desperatly to use the TwoLevelCache and cannot run the first batch
> (note that this batch looks like a big transaction). With 1.0.1, it executes
> in a few seconds, creating 420 * 2 records (not so big). From what i saw, it
> seems that TwoLevelObjectCache reads all materialized objects all time, as
> if it want to check all records againsts all (following relations)! I think
> it is useless reads, accordinf to my opinion. It's definitely not possible
> to have such quantity of instances in memory !!
You have to differ the caching levels. The L2-cache (by default
ObjectDefaultImpl) normally will not cause memory issues, because
SoftReferences are used for caching objects.
http://db.apache.org/ojb/docu/guides/objectcache.html#ObjectCacheDefaultImpl
On the other side to avoid endless loops while materialization of
circular object graphs OJB always cache the whole object graph while
materialization. This is done a "MaterializationCache" (MC) and use hard
references. If the object is fully materialized (e.g. a ProductGroup
with 500 Article objects 1:n reference) the MC will be cleared and push
the objects to the L1 cache (which use SoftReferences again).
So if all 420 objects belong to the same object graph (and there is no
proxy which break the materialization of the circular object graph),
then indeed you can run into memory issues. But this behavior is the
only way to avoid endless loops while materialization of circular object
graphs.
>
> Of course, I tried to break the loading mechanism. First i saw that
> auto-retrieve at "false" can produce fine results, but with the
> incompatibity/disadvantage of ODMG transactions ( following your advices ) i
> tried to keep auto-retrieve and mount CGLIB proxy. The batch nevers ended,
> freezing.
>
Think I don't understand when the issue arise, does the issue arise
while creating/insert of the 420 objects or when you read these objects?
Does the test pass when you decrease the object count or increase the
memory of the JVM? Or does the issue always arise when you use the
TwoLevel cache?
> The only solution who worked has been to first, go back to my old cache
> settings with the ObjectCacheDefaultImpl. and then, yes, first batch runs
> again (faster !!). ouf.
>
strange, I think we should verify whether or not the problem is caused
by a bug in OJB (e.g. endless loop while object materialization). Please
try to run your test with different parameter (see above).
> But after, my problem was about the second batch. It seemed that there is
> problems to read object by identity. I get rid of CGLIB proxies attributes
> (proxy="true" on reference and proxy="dynamic" onto class descriptor) and it
> worked.
>
> So , I'm asking me :
>
> - Does someone really get successful utilization of TwoLevelObjectCacheImpl
> with little transaction as creation/update of 800 records ?
>
The OJB perf-test
http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
run 12 threads handling 500 objects (flat objects without references)
per thread. Running this test with 1000 objects per thread isn't a
problem. This test set JVM <jvmarg value="-Xmx256m"/>, so handling high
object count isn't a problem (except each object itself requires much
memory).
> - Is TwoLevelObjectCache really needed (to avoid dirty reads)?
yep, except if you are using repeatable-read as locking isolation level
and lock (at least read-lock) all objects read and used by the user
http://db.apache.org/ojb/docu/guides/lockmanager.html#Supported+Isolation+Levels
But this has the disadvantage that much more LockNotGrantedExceptions
will occur when concurrent threads operate on the same persistent objects.
> We use to
> work with ObjectCacheDefaultImpl, (re) reading all object before update and
> performance are not so bad.
If you re-read the object after the write lock was established, then it
is nevertheless possible for other threads to read the same object
instance (from the cache) while the tx is running and on rollback the
other threads hold an object with invalid values.
It depends on your application (if this could happen and) if this will
be a problem. If the user re-read the object before update the data
integrity is guaranteed, because the correct object version will be read
and the user will note this (alternative optimistic locking can be used,
this will throw an exception on commit of dirty objects).
>
> - Did i miss something to mount CGLib ability? I set proxy="true" on the
> reference-descriptor and proxy="dynamic" on referenced class.
>
Using a dynamic proxy for the persistent object and a proxy for the 1:1
reference to this object is tautologous. The dynamic proxy replace the
persistent object by a proxy, the reference proxy replace the referenced
object by a proxy instance, thus this will result in a "proxied proxy
object".
In this case you can discard one kind of proxy.
> - I didn't see significant loading chain breaks using CGLib, with
> TwoLevelObjectCache. Tell me i'm wrong and that's the only fact of the cache
> manager..
>
Sorry, I don't understand your question, could you describe more detailed.
regards,
Armin
> Regards
>
>
>
> On 2/6/06, Armin Waibel < arminw@apache.org> wrote:
>> Hi Bruno,
>>
>> Bruno CROS wrote:
>>> About my precedent batch troubles:
>>> In fact, a saw my model loaded from every where with all the actual
>>> auto-retrieve="true", this means, every where !! This means too, that
>>> "back" relations are read too, circling too much. This was the cause of
>> my
>>> OutOfMemoryError.
>>>
>>> My model is a big one with a lot of relations, as complex as you can
>>> imagine.
>>> So, i'm asking me about get rid of those auto-retrieve, to get all
>> objects
>>> reads faster (and avoid a clogging). But i read that for ODMG dev,
>>> precognized settings are auto-retrieve="true" auto-update="none"
>>> auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
>>>
>> In generally the auto-retrieve="true" is mandatory when using the
>> odmg-api. When using it OJB take a snapshot (copy all fields and
>> references to other objects) of each object when it's locked. On commit
>> OJB compare the snapshot with the state of the object on commit. This
>> way OJB can detect changed fields, new or deleted objects in references
>> (1:1, 1:n,m:n).
>> If auto-retrieve is disabled and the object is locked OJB assume that no
>> references exist or the existing ones are deleted although references
>> exist and not be deleted. So this can cause unexpected behavior,
>> particularly with 1:1 references.
>>
>> The easiest way to solve your problem is to use proxy-references. For
>> 1:n and m:n you can use a collection-proxy:
>>
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
>>
>> For 1:1 references you can use proxies too.
>>
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
>> Normally this requires the usage of a interface as persistent object
>> reference field. But when using CGLib based proxies it's not required
>> and you can simply set proxy="true" without any changes in your source
>> code.
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
>>
>>
>> If you can't use proxies (e.g. in a 3-tier application) you can disable
>> auto-retrieve if you take care and:
>> - disable implicit locking in generally
>> - carefully lock all objects before change it (new objects too)
>> - before you lock an object (for update, delete,...) retrieve the
>> references of that object using method
>> PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
>>
>>
>>> At start, I saw that setting auto-retrieve to "true" everywhere wasn't
>>> solution, but all transaction and batch processes were working fine (
>> until
>>> 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
>>> But with a
>>> little doubt, I tell to all the dev team to avoid as possible the read
>> by
>>> iterating collections without any reasons, prefering ReportQuery (single
>>> shot) and direct object queries.
>>> Is that the good way to have a fast and robust application ?
>> ...yep. The fastest way to lookup a single object by PK is to use
>> PB.getObjectByIdentity(...). If a cache is used this method doesn't
>> require a DB round trip in most cases.
>> http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>>
>>
>>
>>> Another thing, does defaultPersistenceBroker always return the tx broker
>>> when tx has begun (in 1.0.4)? That's very important.
>>> I hope that i have not to implement with TransactionExt.getBroker method.
>>> All ours reads are with defaultPersistenceBroker.
>> If you call PBF.defaultPB(...) OJB always return a separate PB instance
>> (of the 'default' connection defined in jdbc-connection-descriptor) from
>> the PB-pool using it's own connection.
>> TransactionExt.getBroker always returns the current used PB instance (of
>> tx). This behavior never changed.
>> If you only do read operations with the separate PB instance you will
>> not run into problems, except if you call tx.flush(). In that case the
>> objects written to database will not be noticed by the separate PB
>> instance (until tx.commit - with default DB isolation level).
>>
>> regards,
>> Armin
>>
>>> Thank you all in advance.
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Auto-retrieve or not ?
Posted by Bruno CROS <br...@gmail.com>.
(to Armin) OK. Really do appreciate all your advices . Thanks again.
Here's my situation, resulting of a migration from 1.0.1 to 1.0.4 (within
ojb-blank) :
- I tried desperatly to use the TwoLevelCache and cannot run the first batch
(note that this batch looks like a big transaction). With 1.0.1, it executes
in a few seconds, creating 420 * 2 records (not so big). From what i saw, it
seems that TwoLevelObjectCache reads all materialized objects all time, as
if it want to check all records againsts all (following relations)! I think
it is useless reads, accordinf to my opinion. It's definitely not possible
to have such quantity of instances in memory !!
Of course, I tried to break the loading mechanism. First i saw that
auto-retrieve at "false" can produce fine results, but with the
incompatibity/disadvantage of ODMG transactions ( following your advices ) i
tried to keep auto-retrieve and mount CGLIB proxy. The batch nevers ended,
freezing.
The only solution who worked has been to first, go back to my old cache
settings with the ObjectCacheDefaultImpl. and then, yes, first batch runs
again (faster !!). ouf.
But after, my problem was about the second batch. It seemed that there is
problems to read object by identity. I get rid of CGLIB proxies attributes
(proxy="true" on reference and proxy="dynamic" onto class descriptor) and it
worked.
So , I'm asking me :
- Does someone really get successful utilization of TwoLevelObjectCacheImpl
with little transaction as creation/update of 800 records ?
- Is TwoLevelObjectCache really needed (to avoid dirty reads)? We use to
work with ObjectCacheDefaultImpl, (re) reading all object before update and
performance are not so bad.
- Did i miss something to mount CGLib ability? I set proxy="true" on the
reference-descriptor and proxy="dynamic" on referenced class.
- I didn't see significant loading chain breaks using CGLib, with
TwoLevelObjectCache. Tell me i'm wrong and that's the only fact of the cache
manager..
Regards
On 2/6/06, Armin Waibel < arminw@apache.org> wrote:
>
> Hi Bruno,
>
> Bruno CROS wrote:
> > About my precedent batch troubles:
> > In fact, a saw my model loaded from every where with all the actual
> > auto-retrieve="true", this means, every where !! This means too, that
> > "back" relations are read too, circling too much. This was the cause of
> my
> > OutOfMemoryError.
> >
> > My model is a big one with a lot of relations, as complex as you can
> > imagine.
> > So, i'm asking me about get rid of those auto-retrieve, to get all
> objects
> > reads faster (and avoid a clogging). But i read that for ODMG dev,
> > precognized settings are auto-retrieve="true" auto-update="none"
> > auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
> >
>
> In generally the auto-retrieve="true" is mandatory when using the
> odmg-api. When using it OJB take a snapshot (copy all fields and
> references to other objects) of each object when it's locked. On commit
> OJB compare the snapshot with the state of the object on commit. This
> way OJB can detect changed fields, new or deleted objects in references
> (1:1, 1:n,m:n).
> If auto-retrieve is disabled and the object is locked OJB assume that no
> references exist or the existing ones are deleted although references
> exist and not be deleted. So this can cause unexpected behavior,
> particularly with 1:1 references.
>
> The easiest way to solve your problem is to use proxy-references. For
> 1:n and m:n you can use a collection-proxy:
>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
>
> For 1:1 references you can use proxies too.
>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
> Normally this requires the usage of a interface as persistent object
> reference field. But when using CGLib based proxies it's not required
> and you can simply set proxy="true" without any changes in your source
> code.
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
>
>
> If you can't use proxies (e.g. in a 3-tier application) you can disable
> auto-retrieve if you take care and:
> - disable implicit locking in generally
> - carefully lock all objects before change it (new objects too)
> - before you lock an object (for update, delete,...) retrieve the
> references of that object using method
> PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
>
>
> > At start, I saw that setting auto-retrieve to "true" everywhere wasn't
> > solution, but all transaction and batch processes were working fine (
> until
> > 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
> > But with a
> > little doubt, I tell to all the dev team to avoid as possible the read
> by
> > iterating collections without any reasons, prefering ReportQuery (single
> > shot) and direct object queries.
> > Is that the good way to have a fast and robust application ?
>
> ...yep. The fastest way to lookup a single object by PK is to use
> PB.getObjectByIdentity(...). If a cache is used this method doesn't
> require a DB round trip in most cases.
> http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>
>
>
> >
> > Another thing, does defaultPersistenceBroker always return the tx broker
> > when tx has begun (in 1.0.4)? That's very important.
> > I hope that i have not to implement with TransactionExt.getBroker method.
> > All ours reads are with defaultPersistenceBroker.
>
> If you call PBF.defaultPB(...) OJB always return a separate PB instance
> (of the 'default' connection defined in jdbc-connection-descriptor) from
> the PB-pool using it's own connection.
> TransactionExt.getBroker always returns the current used PB instance (of
> tx). This behavior never changed.
> If you only do read operations with the separate PB instance you will
> not run into problems, except if you call tx.flush(). In that case the
> objects written to database will not be noticed by the separate PB
> instance (until tx.commit - with default DB isolation level).
>
> regards,
> Armin
>
> >
> > Thank you all in advance.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
Re: Auto-retrieve or not ?
Posted by Bruno CROS <br...@gmail.com>.
Hi Armin,
I came back on TwoLevelsCache usage and CGLib usage too.
2 news : one good, one bad.
Starting with the good one :
- Using 2L cache and CGLIb with my fisrt batch, actually, it crashed because
of memory issue. You were right. I saw that with xmx256m setup, batch stoped
farther than before. Then i had a look to my code, and decided to change
loop coding. In fact, transaction was open once, and i was doing checkpoint
every record. I changed in a begin/commit transaction every loop. And it
ran well, and fast as never. "Very good", i said to me !!
- Now the bad news : I stopped the server and relaunched it. i launched same
batch and had this error above (at start). It seems that my proxy object
cannot be loaded by a simple getByIdentity !!! Proxy class cannot be found.
So looking to my repository, i saw proxy="true" in class descriptor.Wrong ,
isn't it? I changed to proxy="dynamic" and batch freezes at start now, i
guess on the first proxy read!! . i get rid of proxy="dynamic" setup and
all is working now.
So, now, i known that it's critical to work with a lot of checkpoint and
TwoLevelCacheObject implemenation. That did not occur before. I think we
can warn all users.
So, i want well to use TwoLevelCache, it seems to fast up all, with or
without CGLib.
I cannot setup CGLib well, i setup OJB.properties ( 2 lines ), i set
proxy="dynamic" on the class-descriptor i want to break. And it freezes at
start, reading one object with a getByIdentity i guess.
I think, because CGLib never works on my application, there is something 's
wrong.
Any idea? Which release am i supposed to have ?
Big step this day. Thank you for your help. Again and again. Reaching the
goal...
Regards.
44656 WARN [Thread-5] odmg.ObjectEnvelopeTable - Error while write objects
for tx org.apache.ojb.odmg.TransactionImpl@12062da
org.apache.ojb.broker.PersistenceBrokerException:
org.apache.ojb.broker.metadata.MetadataException:
java.lang.ClassNotFoundException: true
at
org.apache.ojb.broker.core.proxy.CollectionProxyDefaultImpl.loadData(Unknown
Source)
at org.apache.ojb.broker.core.proxy.ListProxyDefaultImpl.loadData(Unknown
Source)
at
org.apache.ojb.broker.core.proxy.CollectionProxyDefaultImpl.getData(Unknown
Source)
at
org.apache.ojb.broker.core.proxy.CollectionProxyDefaultImpl.iterator(Unknown
Source)
at
org.apache.ojb.broker.core.proxy.CollectionProxyDefaultImpl.ojbIterator(Unkn
own Source)
at org.apache.ojb.broker.util.BrokerHelper.getCollectionIterator(Unknown
Source)
at org.apache.ojb.odmg.Image$MultipleRef.referenceProcessing(Unknown
Source)
at org.apache.ojb.odmg.Image.performReferenceDetection(Unknown Source)
at org.apache.ojb.odmg.ObjectEnvelope.markReferenceElements(Unknown Source)
at org.apache.ojb.odmg.ObjectEnvelopeTable.checkAllEnvelopes(Unknown
Source)
at org.apache.ojb.odmg.ObjectEnvelopeTable.writeObjects(Unknown Source)
at org.apache.ojb.odmg.TransactionImpl.doWriteObjects(Unknown Source)
at org.apache.ojb.odmg.TransactionImpl.checkpoint(Unknown Source)
at
fr.gouv.finances.douane.dnsce.mathieu.application.tables.service.UnitesServ.
majUnites(UnitesServ.java:110)
On 2/9/06, Armin Waibel < arminw@apache.org> wrote:
>
> Hi Bruno,
>
> Bruno CROS wrote:
> > Hi Armin, and all
> >
> > Just to clarify my dilemma, i just want to resume my intentions about
> > configuring my repository.
> >
> > Consider that model is done, that 90% of transactions are coded (with
> > reportQuery, and query object and a few getCollections utilizations when
>
> > update), and of course, model is complex as you can imagine (lot of
> > collections ).
> >
> > Well, according to my last experimentations on 1.0.4, i 'm confined
> myself
> > to not use TwoLevelObjectCacheImplementation (i noted a very heavy load
> when
> > retrieving object) and not use reference proxy (CGLib) (processes stops
> > after a while). sorry for that, really.
>
> as I wrote in my previous mail, it would be important to clarify if this
> is a problem relating to your object model (object handling) and less
> memory of the JVM or a bug in OJB.
>
> I did some simple tests and can't detect any problems with object
> materialization and the 2L-Cache. On materialization of objects with
> collection references and proxy=true, OJB only materialize the main
> object and set the proxy placeholder.
>
>
> > So, it appears to me that the only way to reduce loading time, is to
> break
> > some strategical relations, setting up false to many auto-retrieves.
> > Logically, i chose to "break" collection relation
> of central classes (
> > imagine a Customer class referred by 10 others classes.) . This seems to
> > fast up all my retrieves (globally) but i known now i 'm exposed to have
> > unwanted behaviour. And that's i can't well imagine.
> >
> > So, if for my class Customer, all collection descriptors auto-retrieves
> are
> > set to false, after my object Customer1 is loaded, collections are not
> > available to read without doing retrieveAllReference(...) (or its
> friends
> > retrieveReference(....). well.Ok.
> > Now, if idecided to change this object with ODMG ( still without
> > retrieveAllReference), am i exposed to have all existing records
> populating
> > my collections to be deleted? By way of example, imagine Order class,
> > populating my collections orders in Customer. Did all of my orders
> disappear
> > ?
>
> if you update the Customer object or you simply add a new Order objects
> OJB shouldn't cause problem (I did some tests and can't detect
> problems). In the first case OJB will simply update Customer. In the
> second case OJB will insert a new Order objects.
> I tested this behavior only for 1:n references. m:n references should
> behave similar (please test it to verify). 1:1 references will cause
> problems.
>
>
> >
> > A second case : still if Customer class have a collection of Orders. If
> i
> > get Order1 of Customer1 to update it, (still without any
> > retrieveAllReference on Customer1),
> > do i have to expext that something
> > disappear ?
>
> if you only update Order1 - no.
> Anyway I recommend you to add these specific cases to your unit-test
> suite to verify OJB behavior.
>
>
> even if Customer is never locked, just loaded and
> > (re)referenced.
>
> Don't understand, what do you mean with "just loaded and (re)referenced"?
>
> regards,
> Armin
>
> >
> > Thanks very very much for the answers.
> >
> > I hope i will not request you anymore help.
> >
> > Regards.
> >
> >
> >
> >
> >
> >
> >
> > On 2/6/06, Armin Waibel <ar...@apache.org> wrote:
> >> Hi Bruno,
> >>
> >> Bruno CROS wrote:
> >>> About my precedent batch troubles:
> >>> In fact, a saw my model loaded from every where with all the actual
> >>> auto-retrieve="true", this means, every where !! This means too, that
>
> >>> "back" relations are read too, circling too much. This was the cause
> of
> >> my
> >>> OutOfMemoryError.
> >>>
> >>> My model is a big one with a lot of relations, as complex as you can
> >>> imagine.
> >>> So, i'm asking me about get rid of those auto-retrieve, to get all
> >> objects
> >>> reads faster (and avoid a clogging). But i read that for ODMG dev,
> >>> precognized settings are auto-retrieve="true" auto-update="none"
> >>> auto-delete="none". Do i have to absolutely follow this ? If yes, why
> ?
> >>>
> >> In generally the auto-retrieve="true" is mandatory when using the
> >> odmg-api. When using it OJB take a snapshot (copy all fields and
> >> references to other objects) of each object when it's locked. On commit
> >> OJB compare the snapshot with the state of the object on commit. This
> >> way OJB can detect changed fields, new or deleted objects in references
>
> >> (1:1, 1:n,m:n).
> >> If auto-retrieve is disabled and the object is locked OJB assume that
> no
> >> references exist or the existing ones are deleted although references
> >> exist and not be deleted. So this can cause unexpected behavior,
> >> particularly with 1:1 references.
> >>
> >> The easiest way to solve your problem is to use proxy-references. For
> >> 1:n and m:n you can use a collection-proxy:
> >>
> >>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
> >>
> >> For 1:1 references you can use proxies too.
> >>
> >> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
>
> >> Normally this requires the usage of a interface as persistent object
> >> reference field. But when using CGLib based proxies it's not required
> >> and you can simply set proxy="true" without any changes in your source
> >> code.
> >>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
> >>
> >>
> >> If you can't use proxies (e.g. in a 3-tier application) you can disable
> >> auto-retrieve if you take care and:
> >> - disable implicit locking in generally
> >> - carefully lock all objects before change it (new objects too)
> >> - before you lock an object (for update, delete,...) retrieve the
> >> references of that object using method
> >> PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
> >>
> >>
> >>> At start, I saw that setting auto-retrieve to "true" everywhere wasn't
> >>> solution, but all transaction and batch processes were working fine (
> >> until
> >>> 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
> >>> But with a
> >>> little doubt, I tell to all the dev team to avoid as possible the read
> >> by
> >>> iterating collections without any reasons, prefering ReportQuery
> (single
> >>> shot) and direct object queries.
> >>> Is that the good way to have a fast and robust application ?
> >> ...yep. The fastest way to lookup a single object by PK is to use
> >> PB.getObjectByIdentity (...). If a cache is used this method doesn't
> >> require a DB round trip in most cases.
> >> http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>
> >>
> >>
> >>
> >>> Another thing, does defaultPersistenceBroker always return the tx
> broker
> >>> when tx has begun (in 1.0.4)? That's very important.
> >>> I hope that i have not to implement with TransactionExt.getBroker
> method.
> >>> All ours reads are with defaultPersistenceBroker.
> >> If you call PBF.defaultPB(...) OJB always return a separate PB instance
> >> (of the 'default' connection defined in jdbc-connection-descriptor)
> from
> >> the PB-pool using it's own connection.
> >> TransactionExt.getBroker always returns the current used PB instance
> (of
> >> tx). This behavior never changed.
> >> If you only do read operations with the separate PB instance you will
> >> not run into problems, except if you call tx.flush(). In that case the
> >> objects written to database will not be noticed by the separate PB
> >> instance (until tx.commit - with default DB isolation level).
> >>
> >> regards,
> >> Armin
> >>
> >>> Thank you all in advance.
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> >> For additional commands, e-mail: ojb-user-help@db.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
Re: Auto-retrieve or not ?
Posted by Armin Waibel <ar...@apache.org>.
Hi Bruno,
Bruno CROS wrote:
> Hi Armin, and all
>
> Just to clarify my dilemma, i just want to resume my intentions about
> configuring my repository.
>
> Consider that model is done, that 90% of transactions are coded (with
> reportQuery, and query object and a few getCollections utilizations when
> update), and of course, model is complex as you can imagine (lot of
> collections ).
>
> Well, according to my last experimentations on 1.0.4, i 'm confined myself
> to not use TwoLevelObjectCacheImplementation (i noted a very heavy load when
> retrieving object) and not use reference proxy (CGLib) (processes stops
> after a while). sorry for that, really.
as I wrote in my previous mail, it would be important to clarify if this
is a problem relating to your object model (object handling) and less
memory of the JVM or a bug in OJB.
I did some simple tests and can't detect any problems with object
materialization and the 2L-Cache. On materialization of objects with
collection references and proxy=true, OJB only materialize the main
object and set the proxy placeholder.
> So, it appears to me that the only way to reduce loading time, is to break
> some strategical relations, setting up false to many auto-retrieves.
> Logically, i chose to "break" collection relation of central classes (
> imagine a Customer class referred by 10 others classes.) . This seems to
> fast up all my retrieves (globally) but i known now i 'm exposed to have
> unwanted behaviour. And that's i can't well imagine.
>
> So, if for my class Customer, all collection descriptors auto-retrieves are
> set to false, after my object Customer1 is loaded, collections are not
> available to read without doing retrieveAllReference(...) (or its friends
> retrieveReference(....). well.Ok.
> Now, if idecided to change this object with ODMG ( still without
> retrieveAllReference), am i exposed to have all existing records populating
> my collections to be deleted? By way of example, imagine Order class,
> populating my collections orders in Customer. Did all of my orders disappear
> ?
if you update the Customer object or you simply add a new Order objects
OJB shouldn't cause problem (I did some tests and can't detect
problems). In the first case OJB will simply update Customer. In the
second case OJB will insert a new Order objects.
I tested this behavior only for 1:n references. m:n references should
behave similar (please test it to verify). 1:1 references will cause
problems.
>
> A second case : still if Customer class have a collection of Orders. If i
> get Order1 of Customer1 to update it, (still without any
> retrieveAllReference on Customer1),
> do i have to expext that something
> disappear ?
if you only update Order1 - no.
Anyway I recommend you to add these specific cases to your unit-test
suite to verify OJB behavior.
even if Customer is never locked, just loaded and
> (re)referenced.
Don't understand, what do you mean with "just loaded and (re)referenced"?
regards,
Armin
>
> Thanks very very much for the answers.
>
> I hope i will not request you anymore help.
>
> Regards.
>
>
>
>
>
>
>
> On 2/6/06, Armin Waibel <ar...@apache.org> wrote:
>> Hi Bruno,
>>
>> Bruno CROS wrote:
>>> About my precedent batch troubles:
>>> In fact, a saw my model loaded from every where with all the actual
>>> auto-retrieve="true", this means, every where !! This means too, that
>>> "back" relations are read too, circling too much. This was the cause of
>> my
>>> OutOfMemoryError.
>>>
>>> My model is a big one with a lot of relations, as complex as you can
>>> imagine.
>>> So, i'm asking me about get rid of those auto-retrieve, to get all
>> objects
>>> reads faster (and avoid a clogging). But i read that for ODMG dev,
>>> precognized settings are auto-retrieve="true" auto-update="none"
>>> auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
>>>
>> In generally the auto-retrieve="true" is mandatory when using the
>> odmg-api. When using it OJB take a snapshot (copy all fields and
>> references to other objects) of each object when it's locked. On commit
>> OJB compare the snapshot with the state of the object on commit. This
>> way OJB can detect changed fields, new or deleted objects in references
>> (1:1, 1:n,m:n).
>> If auto-retrieve is disabled and the object is locked OJB assume that no
>> references exist or the existing ones are deleted although references
>> exist and not be deleted. So this can cause unexpected behavior,
>> particularly with 1:1 references.
>>
>> The easiest way to solve your problem is to use proxy-references. For
>> 1:n and m:n you can use a collection-proxy:
>>
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
>>
>> For 1:1 references you can use proxies too.
>>
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
>> Normally this requires the usage of a interface as persistent object
>> reference field. But when using CGLib based proxies it's not required
>> and you can simply set proxy="true" without any changes in your source
>> code.
>> http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
>>
>>
>> If you can't use proxies (e.g. in a 3-tier application) you can disable
>> auto-retrieve if you take care and:
>> - disable implicit locking in generally
>> - carefully lock all objects before change it (new objects too)
>> - before you lock an object (for update, delete,...) retrieve the
>> references of that object using method
>> PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
>>
>>
>>> At start, I saw that setting auto-retrieve to "true" everywhere wasn't
>>> solution, but all transaction and batch processes were working fine (
>> until
>>> 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
>>> But with a
>>> little doubt, I tell to all the dev team to avoid as possible the read
>> by
>>> iterating collections without any reasons, prefering ReportQuery (single
>>> shot) and direct object queries.
>>> Is that the good way to have a fast and robust application ?
>> ...yep. The fastest way to lookup a single object by PK is to use
>> PB.getObjectByIdentity(...). If a cache is used this method doesn't
>> require a DB round trip in most cases.
>> http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>>
>>
>>
>>> Another thing, does defaultPersistenceBroker always return the tx broker
>>> when tx has begun (in 1.0.4)? That's very important.
>>> I hope that i have not to implement with TransactionExt.getBroker method.
>>> All ours reads are with defaultPersistenceBroker.
>> If you call PBF.defaultPB(...) OJB always return a separate PB instance
>> (of the 'default' connection defined in jdbc-connection-descriptor) from
>> the PB-pool using it's own connection.
>> TransactionExt.getBroker always returns the current used PB instance (of
>> tx). This behavior never changed.
>> If you only do read operations with the separate PB instance you will
>> not run into problems, except if you call tx.flush(). In that case the
>> objects written to database will not be noticed by the separate PB
>> instance (until tx.commit - with default DB isolation level).
>>
>> regards,
>> Armin
>>
>>> Thank you all in advance.
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Auto-retrieve or not ?
Posted by Bruno CROS <br...@gmail.com>.
Hi Armin, and all
Just to clarify my dilemma, i just want to resume my intentions about
configuring my repository.
Consider that model is done, that 90% of transactions are coded (with
reportQuery, and query object and a few getCollections utilizations when
update), and of course, model is complex as you can imagine (lot of
collections ).
Well, according to my last experimentations on 1.0.4, i 'm confined myself
to not use TwoLevelObjectCacheImplementation (i noted a very heavy load when
retrieving object) and not use reference proxy (CGLib) (processes stops
after a while). sorry for that, really.
So, it appears to me that the only way to reduce loading time, is to break
some strategical relations, setting up false to many auto-retrieves.
Logically, i chose to "break" collection relation of central classes (
imagine a Customer class referred by 10 others classes.) . This seems to
fast up all my retrieves (globally) but i known now i 'm exposed to have
unwanted behaviour. And that's i can't well imagine.
So, if for my class Customer, all collection descriptors auto-retrieves are
set to false, after my object Customer1 is loaded, collections are not
available to read without doing retrieveAllReference(...) (or its friends
retrieveReference(....). well.Ok.
Now, if idecided to change this object with ODMG ( still without
retrieveAllReference), am i exposed to have all existing records populating
my collections to be deleted? By way of example, imagine Order class,
populating my collections orders in Customer. Did all of my orders disappear
?
A second case : still if Customer class have a collection of Orders. If i
get Order1 of Customer1 to update it, (still without any
retrieveAllReference on Customer1), do i have to expext that something
disappear ? even if Customer is never locked, just loaded and
(re)referenced.
Thanks very very much for the answers.
I hope i will not request you anymore help.
Regards.
On 2/6/06, Armin Waibel <ar...@apache.org> wrote:
>
> Hi Bruno,
>
> Bruno CROS wrote:
> > About my precedent batch troubles:
> > In fact, a saw my model loaded from every where with all the actual
> > auto-retrieve="true", this means, every where !! This means too, that
> > "back" relations are read too, circling too much. This was the cause of
> my
> > OutOfMemoryError.
> >
> > My model is a big one with a lot of relations, as complex as you can
> > imagine.
> > So, i'm asking me about get rid of those auto-retrieve, to get all
> objects
> > reads faster (and avoid a clogging). But i read that for ODMG dev,
> > precognized settings are auto-retrieve="true" auto-update="none"
> > auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
> >
>
> In generally the auto-retrieve="true" is mandatory when using the
> odmg-api. When using it OJB take a snapshot (copy all fields and
> references to other objects) of each object when it's locked. On commit
> OJB compare the snapshot with the state of the object on commit. This
> way OJB can detect changed fields, new or deleted objects in references
> (1:1, 1:n,m:n).
> If auto-retrieve is disabled and the object is locked OJB assume that no
> references exist or the existing ones are deleted although references
> exist and not be deleted. So this can cause unexpected behavior,
> particularly with 1:1 references.
>
> The easiest way to solve your problem is to use proxy-references. For
> 1:n and m:n you can use a collection-proxy:
>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
>
> For 1:1 references you can use proxies too.
>
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
> Normally this requires the usage of a interface as persistent object
> reference field. But when using CGLib based proxies it's not required
> and you can simply set proxy="true" without any changes in your source
> code.
> http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
>
>
> If you can't use proxies (e.g. in a 3-tier application) you can disable
> auto-retrieve if you take care and:
> - disable implicit locking in generally
> - carefully lock all objects before change it (new objects too)
> - before you lock an object (for update, delete,...) retrieve the
> references of that object using method
> PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
>
>
> > At start, I saw that setting auto-retrieve to "true" everywhere wasn't
> > solution, but all transaction and batch processes were working fine (
> until
> > 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
> > But with a
> > little doubt, I tell to all the dev team to avoid as possible the read
> by
> > iterating collections without any reasons, prefering ReportQuery (single
> > shot) and direct object queries.
> > Is that the good way to have a fast and robust application ?
>
> ...yep. The fastest way to lookup a single object by PK is to use
> PB.getObjectByIdentity(...). If a cache is used this method doesn't
> require a DB round trip in most cases.
> http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>
>
>
> >
> > Another thing, does defaultPersistenceBroker always return the tx broker
> > when tx has begun (in 1.0.4)? That's very important.
> > I hope that i have not to implement with TransactionExt.getBroker method.
> > All ours reads are with defaultPersistenceBroker.
>
> If you call PBF.defaultPB(...) OJB always return a separate PB instance
> (of the 'default' connection defined in jdbc-connection-descriptor) from
> the PB-pool using it's own connection.
> TransactionExt.getBroker always returns the current used PB instance (of
> tx). This behavior never changed.
> If you only do read operations with the separate PB instance you will
> not run into problems, except if you call tx.flush(). In that case the
> objects written to database will not be noticed by the separate PB
> instance (until tx.commit - with default DB isolation level).
>
> regards,
> Armin
>
> >
> > Thank you all in advance.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
Re: Auto-retrieve or not ?
Posted by Armin Waibel <ar...@apache.org>.
Hi Bruno,
Bruno CROS wrote:
> About my precedent batch troubles:
> In fact, a saw my model loaded from every where with all the actual
> auto-retrieve="true", this means, every where !! This means too, that
> "back" relations are read too, circling too much. This was the cause of my
> OutOfMemoryError.
>
> My model is a big one with a lot of relations, as complex as you can
> imagine.
> So, i'm asking me about get rid of those auto-retrieve, to get all objects
> reads faster (and avoid a clogging). But i read that for ODMG dev,
> precognized settings are auto-retrieve="true" auto-update="none"
> auto-delete="none". Do i have to absolutely follow this ? If yes, why ?
>
In generally the auto-retrieve="true" is mandatory when using the
odmg-api. When using it OJB take a snapshot (copy all fields and
references to other objects) of each object when it's locked. On commit
OJB compare the snapshot with the state of the object on commit. This
way OJB can detect changed fields, new or deleted objects in references
(1:1, 1:n,m:n).
If auto-retrieve is disabled and the object is locked OJB assume that no
references exist or the existing ones are deleted although references
exist and not be deleted. So this can cause unexpected behavior,
particularly with 1:1 references.
The easiest way to solve your problem is to use proxy-references. For
1:n and m:n you can use a collection-proxy:
http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Single+Proxy+for+a+Whole+Collection
For 1:1 references you can use proxies too.
http://db.apache.org/ojb/docu/guides/basic-technique.html#Using+a+Proxy+for+a+Reference
Normally this requires the usage of a interface as persistent object
reference field. But when using CGLib based proxies it's not required
and you can simply set proxy="true" without any changes in your source code.
http://db.apache.org/ojb/docu/guides/basic-technique.html#Customizing+the+proxy+mechanism
If you can't use proxies (e.g. in a 3-tier application) you can disable
auto-retrieve if you take care and:
- disable implicit locking in generally
- carefully lock all objects before change it (new objects too)
- before you lock an object (for update, delete,...) retrieve the
references of that object using method
PersistenceBroker.retrieveAllReferences(obj)/retrieveReference(...).
> At start, I saw that setting auto-retrieve to "true" everywhere wasn't
> solution, but all transaction and batch processes were working fine ( until
> 1.0.4. ), with autoretrieve on all n relations (yes!). Chance !!
> But with a
> little doubt, I tell to all the dev team to avoid as possible the read by
> iterating collections without any reasons, prefering ReportQuery (single
> shot) and direct object queries.
> Is that the good way to have a fast and robust application ?
...yep. The fastest way to lookup a single object by PK is to use
PB.getObjectByIdentity(...). If a cache is used this method doesn't
require a DB round trip in most cases.
http://db.apache.org/ojb/docu/tutorials/pb-tutorial.html#Find+object+by+primary+key
>
> Another thing, does defaultPersistenceBroker always return the tx broker
> when tx has begun (in 1.0.4)? That's very important.
> I hope that i have not to implement with TransactionExt.getBroker method.
> All ours reads are with defaultPersistenceBroker.
If you call PBF.defaultPB(...) OJB always return a separate PB instance
(of the 'default' connection defined in jdbc-connection-descriptor) from
the PB-pool using it's own connection.
TransactionExt.getBroker always returns the current used PB instance (of
tx). This behavior never changed.
If you only do read operations with the separate PB instance you will
not run into problems, except if you call tx.flush(). In that case the
objects written to database will not be noticed by the separate PB
instance (until tx.commit - with default DB isolation level).
regards,
Armin
>
> Thank you all in advance.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org