Fwd to teh dev ML : Re: Could I know the progress of LMDB backend for ApacheDS and I want have a try on it

[Emmanuel Lécharny <el...@gmail.com>]

hi Emmanuel,

  Thank you so much for the quick reply.

> check on
https://lists.apache.org/thread.html/98d4a01b66caf0d39af4c437920283a5fa25ade75fe9f4262fa2bae1@%3Cdev.directory.apache.org%3E)

  I have read this report. Excellent job! Impressive results! Look forward to using Mavibot as production soon.

> No progress so far, due to time compression (new born baby, day job..)

  Congratulations to you to have a baby! :) I will start to implement the LMDB backend from tomorrow immediately. And will post results to ApacheDS community when I complete and have performance test results. I am so glad to have the opportunity to contribute.

> LBDM brings zero-copy, because it's based on Memory Mapped File, but
OTOH you won't be able to benefit from it as much as you'd like, due to
the need for serialization/deserialization. Basically, you will need to
add an intermediary cache, otherwise you'll get degraded performances
because when you fetch a page from disk, you will have to deserialize
the whole content, even if you don't need more than just a key and value.

  Appreciate for this advice. I haven't been aware of the effect of this cache for LMDB before. I will test and use cache to optimize this.

Thanks,
Zack Liu




________________________________
From: Emmanuel Lécharny <el...@gmail.com>
Sent: Friday, March 10, 2017 4:32 PM
To: Zack Liu
Subject: Re: Could I know the progress of LMDB backend for ApacheDS and I want have a try on it

Hi,


Le 10/03/2017 à 16:38, Zack Liu a écrit :
> Hi Emmanuel and Zheng,
>
>   Recently I am using ApacheDS as the storage of my project but worried about the DB backend of it. The index of JDBM may be corrupted somehow and it is seems hard to resolve that issue. Another concern is that it is too slow to turn on the syncOnWrite or facing the risk of loss of data when crashing or power off if turn off the syncOnWrite option. Mavibot maybe a choice but it is not for production yet currently.

I have resumed teh work on Mavibot one month ago. Addition are now
working, deletion is under work. This version will support transactions
across multiple B-trees and operations (ie, you will be able to wait
before committing the data to disk). Currently, I have conducted some
tests where transactions are committed across 3 B-trees and committed
after various number of ops (from 1 opeeration to 5000 batched
operations : check on
https://lists.apache.org/thread.html/98d4a01b66caf0d39af4c437920283a5fa25ade75fe9f4262fa2bae1@%3Cdev.directory.apache.org%3E)
>
>   So I consider to implement the LMDB partition and index myself to provide a stable DB backend.
Good ! We would welcome such an implementation !

>  I read this email threads https://www.mail-archive.com/dev@directory.apache.org/msg48554.html and know that you guys want to have a try to implement LMDB backend for ApacheDS. So how about the progress? I want to have a try on LMDB and it really helps if there is some work I can leverage.

No progress so far, due to time compression (new born baby, day job..)
>
>   LMDB claims it supports “Multiple sub-databases may be created with transactions covering all sub-databases”.

It does.

> So maybe the locks in the AbstractBTreePartition can be removed and use LMDB transaction instead.

Absolutely. This is also true for Mavibot, and it's also an absolute
requirement for ApacheDS (and FTR it's the cause of the database
corruption...)


> It seem there’s a hope that LMDB may provides faster read/write speed and a consistent DB backend.

I would say it's a certainty. Mavibot will also be faster than JDBM :
the tests we conducted 2 years ago with teh Mavibot partition shown that
additions were 5 times faster, and searches were twice faster (from teh
top of my head).

LBDM brings zero-copy, because it's based on Memory Mapped File, but
OTOH you won't be able to benefit from it as much as you'd like, due to
the need for serialization/deserialization. Basically, you will need to
add an intermediary cache, otherwise you'll get degraded performances
because when you fetch a page from disk, you will have to deserialize
the whole content, even if you don't need more than just a key and value.

But that is just optimisation.

Feel free to ask more if needed !

--
Emmanuel Lecharny

Symas.com
directory.apache.org