[Mavibot] Heads up

[Emmanuel Lécharny <el...@gmail.com>]

Hi !

I fnally found some time in the train those last two days (nothing beats
7 hours in a train when it comes to code isolated from life and its
contengencies ;-) to complete what I was working for. Here is a quick
status :
o we now have two BTree, in-memory and managed
o the keys and values are stored in Holders (KeyHolder and ValueHolder)
which store the raw representaion (ie, as byte[]). This save a hell lot
of time when we just load a page, read one value and discard the page
from the cache
o a cache is now used to store Nodes and Leaves
o when we have multiple values for a key, we store them as an array of
values if we have less than a given number (the threshold), then we
switch to a sub-btree
o currently, we switch to sub-btrees when we have more than 8 values

It works, I was able to use it in ApacheDS, and I was able to inject 100
000 entries in the server in around 30 mins (around 55 addition per
second).

There is room for improvements though :
o when we switch to sub-btree, we inject all the array's values one by
one, and the sub-btree is written on disk as many times as we have
values. This is sub-optimal
o the findPos(K) method is called too many times (twice more than needed)
o the sub-btree are not cleaned when we don't need them anymore
o we need more tests, especially performance tests
o the code needs a bit of cleanup, so does the Javadoc

I will commit what I was working on in the last 3 weeks, and we may cut
a 0.2 just after.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: [Mavibot] Heads up

[Kiran Ayyagari <ka...@apache.org>]

perfect, thanks for the heads up


On Sat, Oct 26, 2013 at 1:03 AM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi !
>
> I fnally found some time in the train those last two days (nothing beats
> 7 hours in a train when it comes to code isolated from life and its
> contengencies ;-) to complete what I was working for. Here is a quick
> status :
> o we now have two BTree, in-memory and managed
> o the keys and values are stored in Holders (KeyHolder and ValueHolder)
> which store the raw representaion (ie, as byte[]). This save a hell lot
> of time when we just load a page, read one value and discard the page
> from the cache
> o a cache is now used to store Nodes and Leaves
> o when we have multiple values for a key, we store them as an array of
> values if we have less than a given number (the threshold), then we
> switch to a sub-btree
> o currently, we switch to sub-btrees when we have more than 8 values
>
> It works, I was able to use it in ApacheDS, and I was able to inject 100
> 000 entries in the server in around 30 mins (around 55 addition per
> second).
>
> There is room for improvements though :
> o when we switch to sub-btree, we inject all the array's values one by
> one, and the sub-btree is written on disk as many times as we have
> values. This is sub-optimal
> o the findPos(K) method is called too many times (twice more than needed)
> o the sub-btree are not cleaned when we don't need them anymore
> o we need more tests, especially performance tests
> o the code needs a bit of cleanup, so does the Javadoc
>
> I will commit what I was working on in the last 3 weeks, and we may cut
> a 0.2 just after.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com