You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@directory.apache.org by Florent Gratta <fl...@sicap-france.com> on 2008/10/22 11:20:42 UTC

Usage of JDBM : Experience feedbacks

Hi All,

I am currently Project Lead at Sicap, a Swiss company which designs
softwares for mobile operators and the mobile network world in general.

We are going to use a BTree library to index entries of csv files. We
have identified JDBM library (http://jdbm.sourceforge.net/) and we have
seen that you are currently using it in Apache Directory.

As there is no more tracker activity on this project and it seems that
it is the same thing for support, we would like to know if JDBM is
stable:

==> Have you encountered problems when you use this library in
multithreaded mode: Several concurrent inserts at a time?
==> What is the behaviour of the library when we shut down with
emergency? Is it robust? By this way, does BTree file could be
corrupted?
==> Do you encountered performance lack in some cases?
==> Is there some limitations or lacks: Size of the index chain,
cache size etc ...
==> Etc ...

We are very interested on your feedbacks and experiences on JDBM library
usage ?

Thank's a lot,
Kind Regards,
Florent GRATTA

Framework Project Leader, DeVelopment and Operation, Sicap France SAS
Phone +334 37 37 69 96, Fax +334 37 37 69 99, Mobile +336 81 76 58 02
Postal Address: 50 rue de marseille, 69007 Lyon, France
mailto: florent.gratta@sicap-france.com, http://www.sicap.com, Skype:
florent_gratta

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Re: Usage of JDBM : Experience feedbacks

Posted by David Boreham <da...@bozemanpass.com>.

Howard Chu wrote:
> I find that quite difficult to believe. On a properly configured 
> installation we never see corruption or data loss. We've even tested 
> on machines that crashed due to failing DRAMs and the database 
> (containing over 1 terabyte of data) still recovered in under 2 
> minutes on the subsequent reboot (after replacement of bad memory).
+1 on that.

Re: Usage of JDBM : Experience feedbacks

Posted by Emmanuel Lecharny <el...@gmail.com>.

Howard Chu wrote:
> Robert Mertens wrote:
>> Hi Florent,
>> I am using the JDBM as a part of the Apache Directory Server. I am
>> switching from OpenLDAP because it's Berkeley did not always shut down
>> properly.
>
> I find that quite difficult to believe. On a properly configured 
> installation we never see corruption or data loss. We've even tested 
> on machines that crashed due to failing DRAMs and the database 
> (containing over 1 terabyte of data) still recovered in under 2 
> minutes on the subsequent reboot (after replacement of bad memory).
>
> Ever since BerkeleyDB 4.2 came out, we've had no data corruption 
> issues with BerkeleyDB...
I must say that I trust Howard on that ...
>
>> The JDBM seems to do the job, but does not seemt to handle
>> emergency shutdowns very well(or is it the Apache Directory Server?).
It depends on many factors. If you don't select a sync on write (ie, 
differing writes for better performances), you may lose data. Otherwise, 
a crashed JDBM base may need some recovery. We have to investigate this.
>> The BTree was corrupted when shutting down the machine it is running on
>> when we, at the same time, performed actions against the Directory
>> Server where we deleted entries. The result was that the Directory
>> Server could not be restarted again without replacing the JDBM. I am
>> waiting for Apache Directory Server team to implement a failover model
>> for such occasions.
This is something we have in our roadmap for 2.0.

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: Usage of JDBM : Experience feedbacks

Posted by Howard Chu <hy...@symas.com>.

Robert Mertens wrote:
> Hi Florent,
> I am using the JDBM as a part of the Apache Directory Server. I am
> switching from OpenLDAP because it's Berkeley did not always shut down
> properly.

I find that quite difficult to believe. On a properly configured installation 
we never see corruption or data loss. We've even tested on machines that 
crashed due to failing DRAMs and the database (containing over 1 terabyte of 
data) still recovered in under 2 minutes on the subsequent reboot (after 
replacement of bad memory).

Ever since BerkeleyDB 4.2 came out, we've had no data corruption issues with 
BerkeleyDB...

> The JDBM seems to do the job, but does not seemt to handle
> emergency shutdowns very well(or is it the Apache Directory Server?).
> The BTree was corrupted when shutting down the machine it is running on
> when we, at the same time, performed actions against the Directory
> Server where we deleted entries. The result was that the Directory
> Server could not be restarted again without replacing the JDBM. I am
> waiting for Apache Directory Server team to implement a failover model
> for such occasions.
> Best Regards
> Robert Mertens
> +46(0)702-793778
> ------------------------------------------------------------------------
> *From:* Florent Gratta [florent.gratta@sicap-france.com]
> *Sent:* Wednesday, October 22, 2008 11:20 AM
> *To:* dev@directory.apache.org
> *Subject:* Usage of JDBM : Experience feedbacks
>
> Hi All,
>
> I am currently Project Lead at Sicap, a Swiss company which designs
> softwares for mobile operators and the mobile network world in general.
>
> We are going to use a BTree library to index entries of csv files. We
> have identified JDBM library (http://jdbm.sourceforge.net/) and we have
> seen that you are currently using it in Apache Directory.
>
> As there is no more tracker activity on this project and it seems that
> it is the same thing for support, we would like to know if JDBM is stable:
>
> è Have you encountered problems when you use this library in
> multithreaded mode: Several concurrent inserts at a time?
>
> è What is the behaviour of the library when we shut down with emergency?
> Is it robust? By this way, does BTree file could be corrupted?
>
> è Do you encountered performance lack in some cases?
> è Is there some limitations or lacks: Size of the index chain, cache
> size etc ...
> è Etc ...
>
> We are very interested on your feedbacks and experiences on JDBM library
> usage ?

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

RE: Usage of JDBM : Experience feedbacks

Posted by Robert Mertens <ro...@intellus.se>.

Hi Florent,

I am using the JDBM as a part of the Apache Directory Server. I am switching from OpenLDAP because it's Berkeley did not always shut down properly. The JDBM seems to do the job, but does not seemt to handle emergency shutdowns very well(or is it the Apache Directory Server?). The BTree was corrupted when shutting down the machine it is running on when we, at the same time, performed actions against the Directory Server where we deleted entries. The result was that the Directory Server could not be restarted again without replacing the JDBM. I am waiting for Apache Directory Server team to implement a failover model for such occasions.

Best Regards

Robert Mertens
+46(0)702-793778
________________________________
From: Florent Gratta [florent.gratta@sicap-france.com]
Sent: Wednesday, October 22, 2008 11:20 AM
To: dev@directory.apache.org
Subject: Usage of JDBM : Experience feedbacks


Hi All,

I am currently Project Lead at Sicap, a Swiss company which designs softwares for mobile operators and the mobile network world in general.

We are going to use a BTree library to index entries of csv files. We have identified JDBM library (http://jdbm.sourceforge.net/) and we have seen that you are currently using it in Apache Directory.

As there is no more tracker activity on this project and it seems that it is the same thing for support, we would like to know if JDBM is stable:

==>       Have you encountered problems when you use this library in multithreaded mode: Several concurrent inserts at a time?

==>       What is the behaviour of the library when we shut down with emergency? Is it robust? By this way, does BTree file could be corrupted?

==>       Do you encountered performance lack in some cases?
==>       Is there some limitations or lacks: Size of the index chain, cache size etc ...
==>       Etc ...

We are very interested on your feedbacks and experiences on JDBM library usage ?

Thank’s a lot,

Kind Regards,

Florent GRATTA

Framework Project Leader, DeVelopment and Operation, Sicap France SAS

Phone +334 37 37 69 96, Fax +334 37 37 69 99, Mobile +336 81 76 58 02

Postal Address: 50 rue de marseille, 69007 Lyon, France

mailto: florent.gratta@sicap-france.com, http://www.sicap.com, Skype: florent_gratta

________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Re: Usage of JDBM : Experience feedbacks

Posted by Emmanuel Lecharny <el...@gmail.com>.

On Wed, Oct 22, 2008 at 11:20 AM, Florent Gratta
<fl...@sicap-france.com> wrote:
> Hi All,
>
> I am currently Project Lead at Sicap, a Swiss company which designs
> softwares for mobile operators and the mobile network world in general.
>
> We are going to use a BTree library to index entries of csv files. We have
> identified JDBM library (http://jdbm.sourceforge.net/) and we have seen that
> you are currently using it in Apache Directory.
>
> As there is no more tracker activity on this project and it seems that it is
> the same thing for support, we would like to know if JDBM is stable:

There is no activity, mainly because it's damn stable ! It's now 6
years we are using it, and we haven't found a bug... We just have
asked for a couple of modification, just fonctional additions.
>
> è       Have you encountered problems when you use this library in
> multithreaded mode: Several concurrent inserts at a time?

So far, nope.

>
> è       What is the behaviour of the library when we shut down with
> emergency? Is it robust? By this way, does BTree file could be corrupted?

Good question. I guess they could, but nothing a backup can't restore ...

>
> è       Do you encountered performance lack in some cases?

Some very preliminary tests we have done against a major and well none
OSS BTree shows that it's 20 time faster. Now, it's all about how much
memory you have.

> è       Is there some limitations or lacks: Size of the index chain, cache
> size etc ...

None that we can know about. We have injected more than 5 millions 1k
elements into it with something like 5 index, no problem.

> è       Etc ...
>
> We are very interested on your feedbacks and experiences on JDBM library
> usage ?

So far, just a perfect piece of software.


FYI, there is still some activity around JDBM, but it's slow.

Hope it helps.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com