JDBM version & performances

[Emmanuel Lécharny <el...@gmail.com>]

Hi guys,

yesterday, while working on the issue we have when the server is
brutally shutdown, I noticed that we are using a version fo JDBM which
contains some code taht we don't need anymore (the Snapshot code). I
also did some experiment using the TransactionManager, with various
configuration. Here are some findings :

1) configuration impact

When we use the TM, we need to commit() all the modifications and to
sync the transactions peridodically. I tested many diffeent parameters,
like calling commit() for every modification, or once for 2
modifications etc (1, 2, 5,10, 20, 50, 100, 200, 500, 1000...). The very
same for the sync method with the same values (except that I always
start using the same number than the commit - it makes no sense to sync
more often that we commit - ).

The result is that with 200 modifications per commit, and 500 mods per
sync, I get a decent performance (less than 5% slower than not using a
TM at all). I think it's a good compromise as we don't have many pending
modification in case we have a crash.

There is one more parameter I haven't played with, the max number of mod
in a transaction (it should be >= nb mod per commit)

Now, we have to leverage those parameters, which means we should call
the commits() and sync() only when the syncWorker is kicked.

2) JDBM2 contains some extra code, as it's supposed to be a inmemory
MVCC BTree. We don't use this feature, but still, there is some extra
code called. I have conducted some performances on he old JDBM and teh
new one, and I was able to see that the old JDBM is 8 to 10 % faster
than the new one.

Considering that we don't need the Snapshot feature offered by JDBM2, I
think we should switch back to the old JDBM.

I still have to check that we don't have any impact when running
concurrent modifications, but this is just a safety belt here : the way
we fixed the issue is orthogonal.

I'll keep ou inform today.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: JDBM version & performances

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

On 18 janv. 2013, at 10:32, Emmanuel Lécharny <el...@gmail.com> wrote:

> Le 1/18/13 10:24 AM, Kiran Ayyagari a écrit :
>> On Fri, Jan 18, 2013 at 2:24 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
>> 
>>> Hi guys,
>>> 
>>> yesterday, while working on the issue we have when the server is
>>> brutally shutdown, I noticed that we are using a version fo JDBM which
>>> contains some code taht we don't need anymore (the Snapshot code). I
>>> also did some experiment using the TransactionManager, with various
>>> configuration. Here are some findings :
>>> 
>>> 1) configuration impact
>>> 
>>> When we use the TM, we need to commit() all the modifications and to
>>> sync the transactions peridodically. I tested many diffeent parameters,
>>> like calling commit() for every modification, or once for 2
>>> modifications etc (1, 2, 5,10, 20, 50, 100, 200, 500, 1000...). The very
>>> same for the sync method with the same values (except that I always
>>> start using the same number than the commit - it makes no sense to sync
>>> more often that we commit - ).
>>> 
>>> The result is that with 200 modifications per commit, and 500 mods per
>>> sync, I get a decent performance (less than 5% slower than not using a
>>> TM at all). I think it's a good compromise as we don't have many pending
>>> modification in case we have a crash.
>>> 
>>> There is one more parameter I haven't played with, the max number of mod
>>> in a transaction (it should be >= nb mod per commit)
>>> 
>>> Now, we have to leverage those parameters, which means we should call
>>> the commits() and sync() only when the syncWorker is kicked.
> 
>>> it would be nice to do the sync/commit when server is idle(or less loaded)
>> but there are less pending mods than the above threshold
> 
> There is one slight issue with the differed commit/sync : I'm not sure
> it's a thread afe operation. IMO, it should be conducted either while
> processing a modification (ie, counting the per btree modification), or
> using a lock mechanism forbidding any modification to start when he sycn
> is running.
> 
> To be double chcked.
>> 
>>> 2) JDBM2 contains some extra code, as it's supposed to be a inmemory
>>> MVCC BTree. We don't use this feature, but still, there is some extra
>>> code called. I have conducted some performances on he old JDBM and teh
>>> new one, and I was able to see that the old JDBM is 8 to 10 % faster
>>> than the new one.
>>> 
>>> this is good
>>> Considering that we don't need the Snapshot feature offered by JDBM2, I
>>> think we should switch back to the old JDBM.
>>> 
>>> how do we differentiate these 'old' and 'new'?
> 
> I'm updating the jdbm parent's project to reflect teh various versions.
> The 'pld' one will be labelled jdbm1. The new one will be jdbm2.

Everything sounds good to me.

Regards,
Pierre-Arnaud


> -- 
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com 
> 

Re: JDBM version & performances

[Emmanuel Lécharny <el...@gmail.com>]

Le 1/18/13 10:24 AM, Kiran Ayyagari a écrit :
> On Fri, Jan 18, 2013 at 2:24 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
>
>> Hi guys,
>>
>> yesterday, while working on the issue we have when the server is
>> brutally shutdown, I noticed that we are using a version fo JDBM which
>> contains some code taht we don't need anymore (the Snapshot code). I
>> also did some experiment using the TransactionManager, with various
>> configuration. Here are some findings :
>>
>> 1) configuration impact
>>
>> When we use the TM, we need to commit() all the modifications and to
>> sync the transactions peridodically. I tested many diffeent parameters,
>> like calling commit() for every modification, or once for 2
>> modifications etc (1, 2, 5,10, 20, 50, 100, 200, 500, 1000...). The very
>> same for the sync method with the same values (except that I always
>> start using the same number than the commit - it makes no sense to sync
>> more often that we commit - ).
>>
>> The result is that with 200 modifications per commit, and 500 mods per
>> sync, I get a decent performance (less than 5% slower than not using a
>> TM at all). I think it's a good compromise as we don't have many pending
>> modification in case we have a crash.
>>
>> There is one more parameter I haven't played with, the max number of mod
>> in a transaction (it should be >= nb mod per commit)
>>
>> Now, we have to leverage those parameters, which means we should call
>> the commits() and sync() only when the syncWorker is kicked.

>> it would be nice to do the sync/commit when server is idle(or less loaded)
> but there are less pending mods than the above threshold

There is one slight issue with the differed commit/sync : I'm not sure
it's a thread afe operation. IMO, it should be conducted either while
processing a modification (ie, counting the per btree modification), or
using a lock mechanism forbidding any modification to start when he sycn
is running.

To be double chcked.
>
>> 2) JDBM2 contains some extra code, as it's supposed to be a inmemory
>> MVCC BTree. We don't use this feature, but still, there is some extra
>> code called. I have conducted some performances on he old JDBM and teh
>> new one, and I was able to see that the old JDBM is 8 to 10 % faster
>> than the new one.
>>
>> this is good
>> Considering that we don't need the Snapshot feature offered by JDBM2, I
>> think we should switch back to the old JDBM.
>>
>>  how do we differentiate these 'old' and 'new'?

I'm updating the jdbm parent's project to reflect teh various versions.
The 'pld' one will be labelled jdbm1. The new one will be jdbm2.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: JDBM version & performances

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Jan 18, 2013 at 2:24 PM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi guys,
>
> yesterday, while working on the issue we have when the server is
> brutally shutdown, I noticed that we are using a version fo JDBM which
> contains some code taht we don't need anymore (the Snapshot code). I
> also did some experiment using the TransactionManager, with various
> configuration. Here are some findings :
>
> 1) configuration impact
>
> When we use the TM, we need to commit() all the modifications and to
> sync the transactions peridodically. I tested many diffeent parameters,
> like calling commit() for every modification, or once for 2
> modifications etc (1, 2, 5,10, 20, 50, 100, 200, 500, 1000...). The very
> same for the sync method with the same values (except that I always
> start using the same number than the commit - it makes no sense to sync
> more often that we commit - ).
>
> The result is that with 200 modifications per commit, and 500 mods per
> sync, I get a decent performance (less than 5% slower than not using a
> TM at all). I think it's a good compromise as we don't have many pending
> modification in case we have a crash.
>
> There is one more parameter I haven't played with, the max number of mod
> in a transaction (it should be >= nb mod per commit)
>
> Now, we have to leverage those parameters, which means we should call
> the commits() and sync() only when the syncWorker is kicked.
>
> it would be nice to do the sync/commit when server is idle(or less loaded)
but there are less pending mods than the above threshold

> 2) JDBM2 contains some extra code, as it's supposed to be a inmemory
> MVCC BTree. We don't use this feature, but still, there is some extra
> code called. I have conducted some performances on he old JDBM and teh
> new one, and I was able to see that the old JDBM is 8 to 10 % faster
> than the new one.
>
> this is good

> Considering that we don't need the Snapshot feature offered by JDBM2, I
> think we should switch back to the old JDBM.
>
>  how do we differentiate these 'old' and 'new'?

> I still have to check that we don't have any impact when running
> concurrent modifications, but this is just a safety belt here : the way
> we fixed the issue is orthogonal.
>
> I'll keep ou inform today.
>
> thank you for the details

> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com