Mavibot status

[Emmanuel Lécharny <em...@busit.com.INVALID>]

Hi !


it has been a while since I committed something in this repo. I finally 
completed a big rewrite I started 2 years ago.


Basically, it was about supporting cross-B-tree transaction, something 
that was badly missing in the previous release. There are many other 
important changes :

- we don't support multi-values in a B-tree. If one want to stroe such 
things, then the way to go is to create a B-tree aside if the number of 
values is too big.

- The Btree of Btrees (aka BoB) that was used to store all the B-trees 
versions for all the pending transactions has been replaced by a List of 
B-trees. This is simpler to serialize on disk and simpler to manage

- The free pages management is currently disabled, but will soon be 
reestablished. That mean the database file will continuously growth atm.

- We use Caffeine as a internal page cache

- One of the biggest advantage of having a cross-Btree transaction is 
that we can now inject many elements before flushing the data on disk. 
This speeds up insert if we can gather many of those updates into a 
single commit (it can be one or two order of magnitude faster). For 
LDAP, this is interesting because a LDAP update operation generally 
involve the update of 10 or more B-trees, with some of them being 
updated more than one - typically, the RDN index -. But it can also be 
leveraged by the server itself as it now support the LDAP transaction 
extended operation (https://tools.ietf.org/html/rfc5805). It can also be 
used with great benefit for a bulk load, where we commit after 10 of 
thousands entries addition.


There is more to do, and the first step is to add the free page 
management. This is what I'm going to work on.


Thanks !


-- 
-----------------
Emmanuel Lécharny
CTO, Busit


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org

Re: Mavibot status

[Shawn McKinney <sm...@apache.org>]

> On Jun 5, 2020, at 5:19 PM, Emmanuel Lécharny <em...@busit.com.INVALID> wrote:
> 
> it has been a while since I committed something in this repo. I finally completed a big rewrite I started 2 years ago.
> 
> Basically, it was about supporting cross-B-tree transaction, something that was badly missing in the previous release. There are many other important changes :
> 
> - we don't support multi-values in a B-tree. If one want to stroe such things, then the way to go is to create a B-tree aside if the number of values is too big.
> 
> - The Btree of Btrees (aka BoB) that was used to store all the B-trees versions for all the pending transactions has been replaced by a List of B-trees. This is simpler to serialize on disk and simpler to manage
> 
> - The free pages management is currently disabled, but will soon be reestablished. That mean the database file will continuously growth atm.
> 
> - We use Caffeine as a internal page cache
> 
> - One of the biggest advantage of having a cross-Btree transaction is that we can now inject many elements before flushing the data on disk. This speeds up insert if we can gather many of those updates into a single commit (it can be one or two order of magnitude faster). For LDAP, this is interesting because a LDAP update operation generally involve the update of 10 or more B-trees, with some of them being updated more than one - typically, the RDN index -. But it can also be leveraged by the server itself as it now support the LDAP transaction extended operation (https://tools.ietf.org/html/rfc5805). It can also be used with great benefit for a bulk load, where we commit after 10 of thousands entries addition.
> 
> 
> There is more to do, and the first step is to add the free page management. This is what I'm going to work on.
> 

This will lead to 

a. A stable database implementation for apacheDS, i.e. no more data corruptions requiring restores.

b. Better performance on updates.

Long needed in this project.  Thanks for the udpate.  Looking forward to testing this out when ready.

—
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org