You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ojb-user@db.apache.org by Cyril Ledru <cy...@smartjog.com> on 2004/08/12 13:09:24 UTC
Cache problems in mutlithreaded environment
Hi all,
This is my problem : I have an application which launch several threads.
In each thread, there is a call to getCollectionByQuery.
Here is the code of my thread class :
public class LoadThread extends Thread {
public void run() {
PBKey pbkey = new PBKey(...,...,...);
PersistenceBroker pb = null;
try {
pb
=PersistenceBrokerFactory.createPersistenceBroker(pbkey);
Collection c = pb.getCollectionByQuery(new
QueryByCriteria( Employee.class));
} finally {
if (null != pb) {
pb.close();
}
}
}
}
Then i launch this threads :
LoadThread[] ts = new LoadThread[10];
for (int i = 0; i < ts.length; i++) {
ts[i] = new LoadThread();
}
for (int i = 0; i < ts.length; i++) {
ts[i].start();
}
This is quite simple. The problem is the response time are abnormally
high. Usually, only the first call is time consuming because ojb mount
the objects in cache and the next calls should take a lot less time.
But here all the calls are time consuming.
If i launch the same series of threads a second time, the calls take 2
or 3 less time. Same if i put a second call to getCollectionByQuery in
my run method.
So it seems that ojb has a cache for each thread and not a general cache.
Am i using ojb incorrectly or is ojb incapable of caching datas for all
the threads ?
In the second case, is someone knows a way of caching for all the threads ?
Thanks in advance ,
Cyril.
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Armin Waibel <ar...@apache.org>.
Cyril Ledru wrote:
> Thanks again for your answer.
>
> I tried to delay the threads but the result are almost the same.
>
Do you use different delay times for each thread? If all threads wait
e.g 100ms it doesn't change anything, because they will access at same
time again.
> When i told you about the "200ms" it the average time took to retreive
> all employees in the db when the datas are in the cache.
>
> Here, it seems like the datas aren't in a cache...
> Do you know which classes i need to log in order to know if a query
> takes its data in the cache or in the db ?
>
Most cache implementations don't use logging, so you have to change the
source and put an output in ObjectCache#lookup(...) method, e.g.
ObjectCacheDefaultImpl#lookup(...):
...
if(result != null) System.out.println("match: " + oid);
return result;
}
Again, the query is never performed against the cache. First the query
was performed against the DB and then the ResultSet was performed
against the cache. See RsIterator.java.
regards,
Armin
>
>
> Armin Waibel wrote:
>
>>
>>
>> Cyril Ledru wrote:
>>
>>> Thanks for you answer.
>>>
>>> > I think your test case has a few drawbacks and do not reflect a
>>> > production scenario.
>>>
>>> Our clients are asking our datas every 3 sec through a web service.
>>> So this case'll occure a lot i think.
>>>
>>
>> ok, but your test doesn't reflect this. Your test starts ten threads
>> and they all access OJB/DB at the same time. So I think the problem is
>> your DB, because OJB always perform the query to the DB and then check
>> the ResultSet for already cached objects. So I think the test tests
>> the performance of parallel queries against your DB (by query all
>> Employee in DB).
>>
>>> I tried to launch a first dummy call before the others. It reduce a
>>> little the response time but not to much.
>>>
>>
>> ok, first improvement is made ;-)
>>
>>
>>> If i send my requests in a for loop, each request takes 200 ms and if
>>> i do the same but in threads instead of the loop , it takes up to 3
>>> seconds !
>>>
>>
>> hmm, I don't know the size of the returned ResultSet (and the mapping
>> Employee class) so I can't estimate if 200ms is slow. In your test 10
>> threads access at the same time your DB, so 10x200ms will be the best
>> expected result.
>>
>> A more realistic scenario will be
>>
>> public void run()
>> {
>> // each thread obtains objects in a loop e.g 10 times
>> // and use a random timeout (e.g. 10-200ms)
>> for(......)
>> {
>> try{Thread.sleep(timeout);}catch(....)...
>> PBKey pbkey = new PBKey(...,...,...);
>> PersistenceBroker pb = null;
>> try
>> {
>> pb = PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>> Collection c =
>> pb.getCollectionByQuery(new QueryByCriteria(Employee.class));
>> }
>> finally
>> {
>> if (null != pb) pb.close();
>> }
>> }
>> }
>>
>> In this case only a few threads will access the DB at the same time.
>>
>>
>>> > This depends on the used ObjectCache implementation
>>>
>>> I tried all the objectcache but the performance are almost the same.
>>>
>>
>> This prove my "theory" that your DB is the bottleneck.
>>
>> Some days ago I checked in a query performence improvement you can try
>> to replace your local files with these classes
>>
>> /home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RowReaderDefaultImpl.java,v
>>
>> revision 1.30.2.1
>>
>> /home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RsIterator.java,v
>>
>> revision 1.63.2.2
>>
>> This will be part of the upcomming 1.0.1 maintenance release.
>>
>>
>> regards,
>> Armin
>>
>>
>>> > The object you try to materialize will be put to cache when it is
>>> fully
>>> > materialized (e.g. if you set autoRetrieve 'true' to reference
>>> > descriptors they will be load too). So if multiple concurrent threads
>>> > try to lookup the same object (and you use the default cache),
>>> they may
>>> > materialize the object several times because none of the threads
>>> has put
>>> > a fully materialized object to cache.
>>>
>>> My first "dummy" call is in fact the same call to
>>> getCollectionByQuery. So normally all my object should already be in
>>> cache and the next queries sould take a lot less time, aren't they ?
>>>
>>>
>>>
>>>
>>> Armin Waibel wrote:
>>>
>>>> Hi Cyril,
>>>>
>>>> I think your test case has a few drawbacks and do not reflect a
>>>> production scenario.
>>>>
>>>> First time PBF was used OJB parse and create all metadata objects
>>>> (OJB startup phase), this is time consuming and only occur one time
>>>> in OJB runtime. So before you start the LoadThread classes you
>>>> should do a dummy operation on OJB to force OJB startup.
>>>>
>>>> >> If i launch the same series of threads a second time, the calls
>>>> take 2
>>>> >> or 3 less time. Same if i put a second call to
>>>> getCollectionByQuery in
>>>> >> my run method.
>>>> >>
>>>> >> So it seems that ojb has a cache for each thread and not a
>>>> general cache.
>>>>
>>>> This depends on the used ObjectCache implementation
>>>>
>>>> http://db.apache.org/ojb/docu/guides/objectcache.html#Shipped+cache+implementations
>>>>
>>>>
>>>> The object you try to materialize will be put to cache when it is
>>>> fully materialized (e.g. if you set autoRetrieve 'true' to reference
>>>> descriptors they will be load too). So if multiple concurrent
>>>> threads try to lookup the same object (and you use the default
>>>> cache), they may materialize the object several times because none
>>>> of the threads has put a fully materialized object to cache.
>>>>
>>>> By the way, if you use OJB source distribution you can run a
>>>> multi-threaded performance test against (hsql by default) with 'ant
>>>> perf-test'
>>>>
>>>> http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
>>>>
>>>>
>>>> regards,
>>>> Armin
>>>>
>>>>
>>>> Cyril Ledru wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I there a problem with my mail ? Did i forget something ? I am not
>>>>> clear ?
>>>>> Please, tell me, i really need this info.
>>>>>
>>>>> Cyril Ledru wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> This is my problem : I have an application which launch several
>>>>>> threads. In each thread, there is a call to getCollectionByQuery.
>>>>>> Here is the code of my thread class :
>>>>>>
>>>>>> public class LoadThread extends Thread {
>>>>>> public void run() {
>>>>>> PBKey pbkey = new PBKey(...,...,...);
>>>>>> PersistenceBroker pb = null;
>>>>>> try {
>>>>>> pb
>>>>>> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>>>>>>
>>>>>> Collection c = pb.getCollectionByQuery(new
>>>>>> QueryByCriteria( Employee.class));
>>>>>> } finally {
>>>>>> if (null != pb) {
>>>>>> pb.close();
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Then i launch this threads :
>>>>>>
>>>>>> LoadThread[] ts = new LoadThread[10];
>>>>>> for (int i = 0; i < ts.length; i++) {
>>>>>> ts[i] = new LoadThread();
>>>>>> }
>>>>>>
>>>>>> for (int i = 0; i < ts.length; i++) {
>>>>>> ts[i].start();
>>>>>> }
>>>>>>
>>>>>>
>>>>>> This is quite simple. The problem is the response time are
>>>>>> abnormally high. Usually, only the first call is time consuming
>>>>>> because ojb mount the objects in cache and the next calls should
>>>>>> take a lot less time.
>>>>>> But here all the calls are time consuming.
>>>>>>
>>>>>> If i launch the same series of threads a second time, the calls
>>>>>> take 2 or 3 less time. Same if i put a second call to
>>>>>> getCollectionByQuery in my run method.
>>>>>>
>>>>>> So it seems that ojb has a cache for each thread and not a general
>>>>>> cache.
>>>>>>
>>>>>> Am i using ojb incorrectly or is ojb incapable of caching datas
>>>>>> for all the threads ?
>>>>>> In the second case, is someone knows a way of caching for all the
>>>>>> threads ?
>>>>>>
>>>>>> Thanks in advance ,
>>>>>> Cyril.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Cyril Ledru <cy...@smartjog.com>.
Thanks again for your answer.
I tried to delay the threads but the result are almost the same.
When i told you about the "200ms" it the average time took to retreive
all employees in the db when the datas are in the cache.
Here, it seems like the datas aren't in a cache...
Do you know which classes i need to log in order to know if a query
takes its data in the cache or in the db ?
Armin Waibel wrote:
>
>
> Cyril Ledru wrote:
>
>> Thanks for you answer.
>>
>> > I think your test case has a few drawbacks and do not reflect a
>> > production scenario.
>>
>> Our clients are asking our datas every 3 sec through a web service. So
>> this case'll occure a lot i think.
>>
>
> ok, but your test doesn't reflect this. Your test starts ten threads and
> they all access OJB/DB at the same time. So I think the problem is your
> DB, because OJB always perform the query to the DB and then check the
> ResultSet for already cached objects. So I think the test tests the
> performance of parallel queries against your DB (by query all Employee
> in DB).
>
>> I tried to launch a first dummy call before the others. It reduce a
>> little the response time but not to much.
>>
>
> ok, first improvement is made ;-)
>
>
>> If i send my requests in a for loop, each request takes 200 ms and if
>> i do the same but in threads instead of the loop , it takes up to 3
>> seconds !
>>
>
> hmm, I don't know the size of the returned ResultSet (and the mapping
> Employee class) so I can't estimate if 200ms is slow. In your test 10
> threads access at the same time your DB, so 10x200ms will be the best
> expected result.
>
> A more realistic scenario will be
>
> public void run()
> {
> // each thread obtains objects in a loop e.g 10 times
> // and use a random timeout (e.g. 10-200ms)
> for(......)
> {
> try{Thread.sleep(timeout);}catch(....)...
> PBKey pbkey = new PBKey(...,...,...);
> PersistenceBroker pb = null;
> try
> {
> pb = PersistenceBrokerFactory.createPersistenceBroker(pbkey);
> Collection c =
> pb.getCollectionByQuery(new QueryByCriteria(Employee.class));
> }
> finally
> {
> if (null != pb) pb.close();
> }
> }
> }
>
> In this case only a few threads will access the DB at the same time.
>
>
>> > This depends on the used ObjectCache implementation
>>
>> I tried all the objectcache but the performance are almost the same.
>>
>
> This prove my "theory" that your DB is the bottleneck.
>
> Some days ago I checked in a query performence improvement you can try
> to replace your local files with these classes
>
> /home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RowReaderDefaultImpl.java,v
>
> revision 1.30.2.1
>
> /home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RsIterator.java,v
>
> revision 1.63.2.2
>
> This will be part of the upcomming 1.0.1 maintenance release.
>
>
> regards,
> Armin
>
>
>> > The object you try to materialize will be put to cache when it is
>> fully
>> > materialized (e.g. if you set autoRetrieve 'true' to reference
>> > descriptors they will be load too). So if multiple concurrent threads
>> > try to lookup the same object (and you use the default cache), they
>> may
>> > materialize the object several times because none of the threads
>> has put
>> > a fully materialized object to cache.
>>
>> My first "dummy" call is in fact the same call to
>> getCollectionByQuery. So normally all my object should already be in
>> cache and the next queries sould take a lot less time, aren't they ?
>>
>>
>>
>>
>> Armin Waibel wrote:
>>
>>> Hi Cyril,
>>>
>>> I think your test case has a few drawbacks and do not reflect a
>>> production scenario.
>>>
>>> First time PBF was used OJB parse and create all metadata objects
>>> (OJB startup phase), this is time consuming and only occur one time
>>> in OJB runtime. So before you start the LoadThread classes you should
>>> do a dummy operation on OJB to force OJB startup.
>>>
>>> >> If i launch the same series of threads a second time, the calls
>>> take 2
>>> >> or 3 less time. Same if i put a second call to
>>> getCollectionByQuery in
>>> >> my run method.
>>> >>
>>> >> So it seems that ojb has a cache for each thread and not a
>>> general cache.
>>>
>>> This depends on the used ObjectCache implementation
>>>
>>> http://db.apache.org/ojb/docu/guides/objectcache.html#Shipped+cache+implementations
>>>
>>>
>>> The object you try to materialize will be put to cache when it is
>>> fully materialized (e.g. if you set autoRetrieve 'true' to reference
>>> descriptors they will be load too). So if multiple concurrent threads
>>> try to lookup the same object (and you use the default cache), they
>>> may materialize the object several times because none of the threads
>>> has put a fully materialized object to cache.
>>>
>>> By the way, if you use OJB source distribution you can run a
>>> multi-threaded performance test against (hsql by default) with 'ant
>>> perf-test'
>>>
>>> http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
>>>
>>>
>>> regards,
>>> Armin
>>>
>>>
>>> Cyril Ledru wrote:
>>>
>>>> Hi,
>>>>
>>>> I there a problem with my mail ? Did i forget something ? I am not
>>>> clear ?
>>>> Please, tell me, i really need this info.
>>>>
>>>> Cyril Ledru wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> This is my problem : I have an application which launch several
>>>>> threads. In each thread, there is a call to getCollectionByQuery.
>>>>> Here is the code of my thread class :
>>>>>
>>>>> public class LoadThread extends Thread {
>>>>> public void run() {
>>>>> PBKey pbkey = new PBKey(...,...,...);
>>>>> PersistenceBroker pb = null;
>>>>> try {
>>>>> pb
>>>>> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>>>>>
>>>>> Collection c = pb.getCollectionByQuery(new
>>>>> QueryByCriteria( Employee.class));
>>>>> } finally {
>>>>> if (null != pb) {
>>>>> pb.close();
>>>>> }
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> Then i launch this threads :
>>>>>
>>>>> LoadThread[] ts = new LoadThread[10];
>>>>> for (int i = 0; i < ts.length; i++) {
>>>>> ts[i] = new LoadThread();
>>>>> }
>>>>>
>>>>> for (int i = 0; i < ts.length; i++) {
>>>>> ts[i].start();
>>>>> }
>>>>>
>>>>>
>>>>> This is quite simple. The problem is the response time are
>>>>> abnormally high. Usually, only the first call is time consuming
>>>>> because ojb mount the objects in cache and the next calls should
>>>>> take a lot less time.
>>>>> But here all the calls are time consuming.
>>>>>
>>>>> If i launch the same series of threads a second time, the calls
>>>>> take 2 or 3 less time. Same if i put a second call to
>>>>> getCollectionByQuery in my run method.
>>>>>
>>>>> So it seems that ojb has a cache for each thread and not a general
>>>>> cache.
>>>>>
>>>>> Am i using ojb incorrectly or is ojb incapable of caching datas for
>>>>> all the threads ?
>>>>> In the second case, is someone knows a way of caching for all the
>>>>> threads ?
>>>>>
>>>>> Thanks in advance ,
>>>>> Cyril.
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Armin Waibel <ar...@apache.org>.
Cyril Ledru wrote:
> Thanks for you answer.
>
> > I think your test case has a few drawbacks and do not reflect a
> > production scenario.
>
> Our clients are asking our datas every 3 sec through a web service. So
> this case'll occure a lot i think.
>
ok, but your test doesn't reflect this. Your test starts ten threads and
they all access OJB/DB at the same time. So I think the problem is your
DB, because OJB always perform the query to the DB and then check the
ResultSet for already cached objects. So I think the test tests the
performance of parallel queries against your DB (by query all Employee
in DB).
> I tried to launch a first dummy call before the others. It reduce a
> little the response time but not to much.
>
ok, first improvement is made ;-)
> If i send my requests in a for loop, each request takes 200 ms and if i
> do the same but in threads instead of the loop , it takes up to 3 seconds !
>
hmm, I don't know the size of the returned ResultSet (and the mapping
Employee class) so I can't estimate if 200ms is slow. In your test 10
threads access at the same time your DB, so 10x200ms will be the best
expected result.
A more realistic scenario will be
public void run()
{
// each thread obtains objects in a loop e.g 10 times
// and use a random timeout (e.g. 10-200ms)
for(......)
{
try{Thread.sleep(timeout);}catch(....)...
PBKey pbkey = new PBKey(...,...,...);
PersistenceBroker pb = null;
try
{
pb = PersistenceBrokerFactory.createPersistenceBroker(pbkey);
Collection c =
pb.getCollectionByQuery(new QueryByCriteria(Employee.class));
}
finally
{
if (null != pb) pb.close();
}
}
}
In this case only a few threads will access the DB at the same time.
> > This depends on the used ObjectCache implementation
>
> I tried all the objectcache but the performance are almost the same.
>
This prove my "theory" that your DB is the bottleneck.
Some days ago I checked in a query performence improvement you can try
to replace your local files with these classes
/home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RowReaderDefaultImpl.java,v
revision 1.30.2.1
/home/cvs/db-ojb/src/java/org/apache/ojb/broker/accesslayer/RsIterator.java,v
revision 1.63.2.2
This will be part of the upcomming 1.0.1 maintenance release.
regards,
Armin
> > The object you try to materialize will be put to cache when it is fully
> > materialized (e.g. if you set autoRetrieve 'true' to reference
> > descriptors they will be load too). So if multiple concurrent threads
> > try to lookup the same object (and you use the default cache), they may
> > materialize the object several times because none of the threads has put
> > a fully materialized object to cache.
>
> My first "dummy" call is in fact the same call to getCollectionByQuery.
> So normally all my object should already be in cache and the next
> queries sould take a lot less time, aren't they ?
>
>
>
>
> Armin Waibel wrote:
>
>> Hi Cyril,
>>
>> I think your test case has a few drawbacks and do not reflect a
>> production scenario.
>>
>> First time PBF was used OJB parse and create all metadata objects (OJB
>> startup phase), this is time consuming and only occur one time in OJB
>> runtime. So before you start the LoadThread classes you should do a
>> dummy operation on OJB to force OJB startup.
>>
>> >> If i launch the same series of threads a second time, the calls
>> take 2
>> >> or 3 less time. Same if i put a second call to
>> getCollectionByQuery in
>> >> my run method.
>> >>
>> >> So it seems that ojb has a cache for each thread and not a general
>> cache.
>>
>> This depends on the used ObjectCache implementation
>>
>> http://db.apache.org/ojb/docu/guides/objectcache.html#Shipped+cache+implementations
>>
>>
>> The object you try to materialize will be put to cache when it is
>> fully materialized (e.g. if you set autoRetrieve 'true' to reference
>> descriptors they will be load too). So if multiple concurrent threads
>> try to lookup the same object (and you use the default cache), they
>> may materialize the object several times because none of the threads
>> has put a fully materialized object to cache.
>>
>> By the way, if you use OJB source distribution you can run a
>> multi-threaded performance test against (hsql by default) with 'ant
>> perf-test'
>>
>> http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
>>
>>
>> regards,
>> Armin
>>
>>
>> Cyril Ledru wrote:
>>
>>> Hi,
>>>
>>> I there a problem with my mail ? Did i forget something ? I am not
>>> clear ?
>>> Please, tell me, i really need this info.
>>>
>>> Cyril Ledru wrote:
>>>
>>>> Hi all,
>>>>
>>>> This is my problem : I have an application which launch several
>>>> threads. In each thread, there is a call to getCollectionByQuery.
>>>> Here is the code of my thread class :
>>>>
>>>> public class LoadThread extends Thread {
>>>> public void run() {
>>>> PBKey pbkey = new PBKey(...,...,...);
>>>> PersistenceBroker pb = null;
>>>> try {
>>>> pb
>>>> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>>>>
>>>> Collection c = pb.getCollectionByQuery(new
>>>> QueryByCriteria( Employee.class));
>>>> } finally {
>>>> if (null != pb) {
>>>> pb.close();
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>>> Then i launch this threads :
>>>>
>>>> LoadThread[] ts = new LoadThread[10];
>>>> for (int i = 0; i < ts.length; i++) {
>>>> ts[i] = new LoadThread();
>>>> }
>>>>
>>>> for (int i = 0; i < ts.length; i++) {
>>>> ts[i].start();
>>>> }
>>>>
>>>>
>>>> This is quite simple. The problem is the response time are
>>>> abnormally high. Usually, only the first call is time consuming
>>>> because ojb mount the objects in cache and the next calls should
>>>> take a lot less time.
>>>> But here all the calls are time consuming.
>>>>
>>>> If i launch the same series of threads a second time, the calls take
>>>> 2 or 3 less time. Same if i put a second call to
>>>> getCollectionByQuery in my run method.
>>>>
>>>> So it seems that ojb has a cache for each thread and not a general
>>>> cache.
>>>>
>>>> Am i using ojb incorrectly or is ojb incapable of caching datas for
>>>> all the threads ?
>>>> In the second case, is someone knows a way of caching for all the
>>>> threads ?
>>>>
>>>> Thanks in advance ,
>>>> Cyril.
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Cyril Ledru <cy...@smartjog.com>.
Thanks for you answer.
> I think your test case has a few drawbacks and do not reflect a
> production scenario.
Our clients are asking our datas every 3 sec through a web service. So
this case'll occure a lot i think.
I tried to launch a first dummy call before the others. It reduce a
little the response time but not to much.
If i send my requests in a for loop, each request takes 200 ms and if i
do the same but in threads instead of the loop , it takes up to 3 seconds !
> This depends on the used ObjectCache implementation
I tried all the objectcache but the performance are almost the same.
> The object you try to materialize will be put to cache when it is fully
> materialized (e.g. if you set autoRetrieve 'true' to reference
> descriptors they will be load too). So if multiple concurrent threads
> try to lookup the same object (and you use the default cache), they may
> materialize the object several times because none of the threads has put
> a fully materialized object to cache.
My first "dummy" call is in fact the same call to getCollectionByQuery.
So normally all my object should already be in cache and the next
queries sould take a lot less time, aren't they ?
Armin Waibel wrote:
> Hi Cyril,
>
> I think your test case has a few drawbacks and do not reflect a
> production scenario.
>
> First time PBF was used OJB parse and create all metadata objects (OJB
> startup phase), this is time consuming and only occur one time in OJB
> runtime. So before you start the LoadThread classes you should do a
> dummy operation on OJB to force OJB startup.
>
> >> If i launch the same series of threads a second time, the calls take 2
> >> or 3 less time. Same if i put a second call to getCollectionByQuery in
> >> my run method.
> >>
> >> So it seems that ojb has a cache for each thread and not a general
> cache.
>
> This depends on the used ObjectCache implementation
>
> http://db.apache.org/ojb/docu/guides/objectcache.html#Shipped+cache+implementations
>
>
> The object you try to materialize will be put to cache when it is fully
> materialized (e.g. if you set autoRetrieve 'true' to reference
> descriptors they will be load too). So if multiple concurrent threads
> try to lookup the same object (and you use the default cache), they may
> materialize the object several times because none of the threads has put
> a fully materialized object to cache.
>
> By the way, if you use OJB source distribution you can run a
> multi-threaded performance test against (hsql by default) with 'ant
> perf-test'
>
> http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
>
>
> regards,
> Armin
>
>
> Cyril Ledru wrote:
>
>> Hi,
>>
>> I there a problem with my mail ? Did i forget something ? I am not
>> clear ?
>> Please, tell me, i really need this info.
>>
>> Cyril Ledru wrote:
>>
>>> Hi all,
>>>
>>> This is my problem : I have an application which launch several
>>> threads. In each thread, there is a call to getCollectionByQuery.
>>> Here is the code of my thread class :
>>>
>>> public class LoadThread extends Thread {
>>> public void run() {
>>> PBKey pbkey = new PBKey(...,...,...);
>>> PersistenceBroker pb = null;
>>> try {
>>> pb
>>> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>>>
>>> Collection c = pb.getCollectionByQuery(new
>>> QueryByCriteria( Employee.class));
>>> } finally {
>>> if (null != pb) {
>>> pb.close();
>>> }
>>> }
>>> }
>>> }
>>>
>>> Then i launch this threads :
>>>
>>> LoadThread[] ts = new LoadThread[10];
>>> for (int i = 0; i < ts.length; i++) {
>>> ts[i] = new LoadThread();
>>> }
>>>
>>> for (int i = 0; i < ts.length; i++) {
>>> ts[i].start();
>>> }
>>>
>>>
>>> This is quite simple. The problem is the response time are abnormally
>>> high. Usually, only the first call is time consuming because ojb
>>> mount the objects in cache and the next calls should take a lot less
>>> time.
>>> But here all the calls are time consuming.
>>>
>>> If i launch the same series of threads a second time, the calls take
>>> 2 or 3 less time. Same if i put a second call to getCollectionByQuery
>>> in my run method.
>>>
>>> So it seems that ojb has a cache for each thread and not a general
>>> cache.
>>>
>>> Am i using ojb incorrectly or is ojb incapable of caching datas for
>>> all the threads ?
>>> In the second case, is someone knows a way of caching for all the
>>> threads ?
>>>
>>> Thanks in advance ,
>>> Cyril.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Armin Waibel <ar...@apache.org>.
Hi Cyril,
I think your test case has a few drawbacks and do not reflect a
production scenario.
First time PBF was used OJB parse and create all metadata objects (OJB
startup phase), this is time consuming and only occur one time in OJB
runtime. So before you start the LoadThread classes you should do a
dummy operation on OJB to force OJB startup.
>> If i launch the same series of threads a second time, the calls take 2
>> or 3 less time. Same if i put a second call to getCollectionByQuery in
>> my run method.
>>
>> So it seems that ojb has a cache for each thread and not a general
cache.
This depends on the used ObjectCache implementation
http://db.apache.org/ojb/docu/guides/objectcache.html#Shipped+cache+implementations
The object you try to materialize will be put to cache when it is fully
materialized (e.g. if you set autoRetrieve 'true' to reference
descriptors they will be load too). So if multiple concurrent threads
try to lookup the same object (and you use the default cache), they may
materialize the object several times because none of the threads has put
a fully materialized object to cache.
By the way, if you use OJB source distribution you can run a
multi-threaded performance test against (hsql by default) with 'ant
perf-test'
http://db.apache.org/ojb/docu/guides/performance.html#OJB+performance+in+multi-threaded+environments
regards,
Armin
Cyril Ledru wrote:
> Hi,
>
> I there a problem with my mail ? Did i forget something ? I am not clear ?
> Please, tell me, i really need this info.
>
> Cyril Ledru wrote:
>
>> Hi all,
>>
>> This is my problem : I have an application which launch several
>> threads. In each thread, there is a call to getCollectionByQuery.
>> Here is the code of my thread class :
>>
>> public class LoadThread extends Thread {
>> public void run() {
>> PBKey pbkey = new PBKey(...,...,...);
>> PersistenceBroker pb = null;
>> try {
>> pb
>> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>>
>> Collection c = pb.getCollectionByQuery(new
>> QueryByCriteria( Employee.class));
>> } finally {
>> if (null != pb) {
>> pb.close();
>> }
>> }
>> }
>> }
>>
>> Then i launch this threads :
>>
>> LoadThread[] ts = new LoadThread[10];
>> for (int i = 0; i < ts.length; i++) {
>> ts[i] = new LoadThread();
>> }
>>
>> for (int i = 0; i < ts.length; i++) {
>> ts[i].start();
>> }
>>
>>
>> This is quite simple. The problem is the response time are abnormally
>> high. Usually, only the first call is time consuming because ojb mount
>> the objects in cache and the next calls should take a lot less time.
>> But here all the calls are time consuming.
>>
>> If i launch the same series of threads a second time, the calls take 2
>> or 3 less time. Same if i put a second call to getCollectionByQuery in
>> my run method.
>>
>> So it seems that ojb has a cache for each thread and not a general cache.
>>
>> Am i using ojb incorrectly or is ojb incapable of caching datas for
>> all the threads ?
>> In the second case, is someone knows a way of caching for all the
>> threads ?
>>
>> Thanks in advance ,
>> Cyril.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-user-help@db.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org
Re: Cache problems in mutlithreaded environment
Posted by Cyril Ledru <cy...@smartjog.com>.
Hi,
I there a problem with my mail ? Did i forget something ? I am not clear ?
Please, tell me, i really need this info.
Cyril Ledru wrote:
> Hi all,
>
> This is my problem : I have an application which launch several threads.
> In each thread, there is a call to getCollectionByQuery.
> Here is the code of my thread class :
>
> public class LoadThread extends Thread {
> public void run() {
> PBKey pbkey = new PBKey(...,...,...);
> PersistenceBroker pb = null;
> try {
> pb
> =PersistenceBrokerFactory.createPersistenceBroker(pbkey);
>
> Collection c = pb.getCollectionByQuery(new
> QueryByCriteria( Employee.class));
> } finally {
> if (null != pb) {
> pb.close();
> }
> }
> }
> }
>
> Then i launch this threads :
>
> LoadThread[] ts = new LoadThread[10];
> for (int i = 0; i < ts.length; i++) {
> ts[i] = new LoadThread();
> }
>
> for (int i = 0; i < ts.length; i++) {
> ts[i].start();
> }
>
>
> This is quite simple. The problem is the response time are abnormally
> high. Usually, only the first call is time consuming because ojb mount
> the objects in cache and the next calls should take a lot less time.
> But here all the calls are time consuming.
>
> If i launch the same series of threads a second time, the calls take 2
> or 3 less time. Same if i put a second call to getCollectionByQuery in
> my run method.
>
> So it seems that ojb has a cache for each thread and not a general cache.
>
> Am i using ojb incorrectly or is ojb incapable of caching datas for all
> the threads ?
> In the second case, is someone knows a way of caching for all the threads ?
>
> Thanks in advance ,
> Cyril.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-user-help@db.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-user-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-user-help@db.apache.org