[Mavibot] Update, week 5

[Emmanuel Lécharny <el...@gmail.com>]

Hi !


What a week ! Ok I was'nt able to work as much as I would have liked on
Mavibot, but still, there is some progress.

Addition of values have been fixed. I was able to conduct some
performance tests, with various number of elements being added within a
transaction (from 1 to 100 000 per trasactions). I have already
mentinned that gathering updates within a transaction speeds up the
process of adding elements. Here are the numbers :


Delta add    : time it takes to inject 100 000 elements in a B-tree

Delta browse : time it takes to browse the 100 000 elements 1000 times
(so 100 000 000 loops)

File size    : The resulting file size when the 100 000 elements has
been added

Increment    : The number of addition withing one single transaction
(ie, if it's 5, that means we will add 5 elements before committing the
transaction)


100 000 elements, increment 1.
Delta add    : 36.590 (2 732/s)
Delta browse : 12.489
File size    : 1 034 376 704 bytes

100 000 elements, increment 2.
Delta add    : 18.081 (5 530/s)
Delta browse : 11.207
File size    : 508 566 016 bytes

100 000 elements, increment 4.
Delta        :  9.066 (11 030/s)
Delta browse : 11.446
File size    : 252 087 808 bytes

100 000 elements, increment 8.
Delta add    :  4.798 (20 842/s)
Delta browse : 11.379
File size    : 123 848 192 bytes

100 000 elements, increment 10.
Delta add    :  3.879 (25 779/s)
Delta browse : 12.585
File size    : 99 265 024 bytes

100 000 elements, increment 20.
Delta add    :  2.237 (44 702/s)
Delta browse : 11.347
File size    : 51 596 800 bytes

100 000 elements, increment 50.
Delta add    :  1.199 (83 402/s)
Delta browse : 10.961
File size    : 24 276 992 bytes

100 000 elements, increment 100.
Delta add    :  0.899 (111 234/s)
Delta browse : 11.303
File size    : 15 324 160 bytes

100 000 elements, increment 200.
Delta add    :  0.795 (125 786/s)
Delta browse : 10.773
File size    : 11 107 840 bytes

100 000 elements, increment 500.
Delta add    :  0.730 (136 986/s)
File size    : 8 688 640 bytes

100 000 elements, increment 1 000.
Delta add    :  0.795 (126 000/s)
Delta browse : 8 million/s
File size    : 7 903 744 bytes

100 000 elements, increment 2 000.
Delta add    :  1.221 (/s)
File size    : 7 536 640 bytes

100 000 elements, increment 5 000.
Delta add    :  2.444 (/s)
File size    : 7 326 208 bytes

As we can see, we max at around 500 update per transaction, and then it
starts to slow down. I haven't spent time on studying what's going on,
but I suspect the GC is teh culprit.


Anyway, addition is just working fine now. I'm now working on the
deletion, which needs some love : currently, we process pages merge from
the bottom, which requires some extra informations about parents. It's
way better to let the parent deal with merge, and that is what I'm
refactoring atm. It will take a bit fo time ;-)


As soon as add/delete operations are completed, the next step will be to
manage concurrent reads, ie concurrent versions. We need to keep a track
of all the used versions, in order to be able to harvest the dead
versions later on.


I'll keep you posted, of course !


PS : I have committed the code in a branch
(directory/mavibot/branches/single-value/mavibot)

-- 
Emmanuel Lecharny

Symas.com
directory.apache.org