You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2017/01/23 09:12:49 UTC
[Mavibot] Update
Hi guys,
just to inform you that this week-end, I spent time on refactoring the
mavibot code base to implement transaction support. Here is a quick
heads up :
- every read and write will be done in a transaction
- we will have Read transactions and Write trasactions
- only one write transaction can run, the other will have to wait for
the previous one's completion
- I will remove the multi-value support from the core code : it will
have to be handled by the application
The whole idea is that many updates may be applied during one write
transaction, with all the modified pages being hold in memory until the
commit. That will save us a hell lot of disk writes (I woud assume that
nearly 60 to 70 % of the writes will be avoided, speeding up the write
rate by the same amount).
The consequence is that every function will have to take an extra
parameter, a Transaction.
The code should be ready soon, it's not that complex.
The next step will be to implement the reclaimer on top of the
transaction system. I still have to see if this will be the last task
when doing an update (but the pb is that we may wait a long time with
dead pages pending), or if we need a dedicated thread for this (as we do
in the current implementation). All in all, this should not be
complicated, as we already have it working. The only thing that might be
reworked is the algorithm in charge to find the pages that can be
reclaimed. The brutal algorithm just reclaim pages that are older that
the oldest revision in use, a more subtile, but more complex, algorithm
would be able to reclaim pages when a revision is released, even if we
still have older reads going on - we discussed that with Kiran two years
ago... -
The last step will be to implement a correct cache system, because we
currently have to deserialize pages when we fetch them by their offset.
I still have to think about it, but it's an improvement of a working
system, so it may come after the current work (even if it will be
absolutely required on a working system, in rder to have decent
performances. But correctness comes first !)
I hope I can have a running version soon, but I can't set a deadline,
being quite busy atm.
I'll keep you posted.
Thanks !
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: [Mavibot] Update
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi !
Another week-end, another report...
I have made some progress, but not as fast than I would have liked to. I
was able to inject up to 100 000 elements in a B-tree, in a single
transaction. It's at least 5 times faster than with the previous Mavibot
release, thanks to the transaction mechanism, which saves a lot of page
writes. Note that there is no cache in the current version.
I also tried to run a loop with a transaction for each instert, and it
was working up to 1000 inserts, but then I get some errors, that I
wasn't able to get fixed (actually, I fixed a lot of small errors, but I
still have a database corruption to take care of). Even with one update
per transaction, it's still slightly faster than the previous release.
The main issue was linked to the refactoring of page duplication, and
handling of full page splits, as the B-trees grow.
I also ave added a dump utility that expose teh content of the file, to
avoid checking it by calling hexdump on it. A large gain when debugging
the library.
I hope being able to find a bit of time to get the insert fixed soon
enugh. The next step will be to also fix the delete funtion.
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: [Mavibot] Update
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi !
End of the week-end...
FTR here is what looks like the insert operation :
try ( WriteTransaction writeTxn =
recordManager.beginWriteTransaction() )
{
btree.insert( writeTxn, 1L, "1" );
btree.insert( writeTxn, 4L, "4" );
btree.insert( writeTxn, 2L, "2" );
btree.insert( writeTxn, 3L, "3" );
btree.insert( writeTxn, 5L, "5" );
}
As you can see, we use a try-with-resource, where the transaction is
automatically committed, which makes it really easy for the users. Of
course, one can abort the transaction at any moment, and the auto-commit
will jst do nothing in this case.
The extra advantage is that it's harder to forget to close a transaction :-)
Of course, it's still possible to explicitely commit the transaction, if
needed.
One thing that came to my mind is that we don't necessarily need to
detect that we had a crash. Currently, the idea was to save the current
state (ie, the reference to the n-1 revision offsets in the
RecordManagerHeader - the first page in the file -) when we start a txn,
and to replace it with an updated version when we are done with the txn.
If we have a crash in the middle, on restart, we will detect that the
recordManagerHeader is in an intermediary state, and we will be able to
scan the full fine to retrieve unlinked pages - in any case, read and
write are stll possile duting this recovery phase, which is ran
concurrently -. That comes with an extra write for every single txn.
My take is that it's probably less expensive to do this scan on every
start, no matter what, because this is a one-time expense, and this
effort is not impacting reads and writes anyway. That would save one
extra write per write transaction. In any case, the data will always be
consistant, no matter what.
More next week !
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: [Mavibot] Update
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi guys,
week-end, so good to have time to get things done :-)
So mavibot + txn is making progress. I now have multiple writes gathered
into one single update on disk. That saves a lot of writes. Typically,
in a test, I start a twn, insert 5 values, and commit the txn, and I
just have one new version being written, saving a global 25 page writes
(5 instead of 30).
All the b-trees being updated (test, BOB and CPB) have the same
revisions. All the b-trees are correctly serialized on disk, although
there are some optimisation to be made for the CPB content (see later)
Now, I had some more thoughts on the CPB b-tree (the CopiedPagesBtree).
We store a list of each pages that have been copied by a given revision,
for each specific B-tree. Two things :
- as this tree contains page that have been copied, and that will be
removed when a specific revision will be phased out, there is no mean in
keeping the original b-tree name : we don't care about ths name when it
comes to move the page in the free-list page. That will save a bit of
room, and we will have less elements in the b-tree
- but overall, we may not need to keep this b-tree on disk. If the
system crashes, the old revisions are dead anyway, and teh associated
pages are going to be moved to the free-page list anyway. The only
reason we have this CPB written to disk is to be able to know which
pages to remove when the system restarts. A faster solution would simply
be to keep this b-tree in memory (and to use a list instead of a B-tree.
We always append new copied page references at the end of this list, and
we always remove them from the beginning of the list. It's a small set
of data, as soon as we don't keeprunning txn for ever (if 30 pages are
copied during a txn, we are talking about 30*8 +8 bytes, around 256
bytes counting the reference to the next elem in the list. If we keep 1
million txn, that will be 256 Mb in memory. Unlikely to happen...). If
we keep this data structure in memory, we will save some disk writes,
and recovering from a crash will just be a mmater for the reclaimer to
parse the entire file, pointing the unused pages, and add them in the
free-list page. It should not take a lt of time, and should not block
any read or writes, so it can be done in a dedicated thread.
Last, not least, the free-page-list might also be kept in memory, but
this is a bit more a constraint, because that would defer any writes to
the moment the reclaimer has controlled all the unused pages on the
disk. If we don't want to block teh writes, that will imply any update
done until this check is completed will get some new pages from the end
of the file, elading to some growing file. Not necessarly a big issue,
but still...
Anyway, I have to complete the Node serialization (I just completed the
leaf), to use the txn system in all the existing tests, and to reboot
the reclaimer. That will be partly for today and partly for next week-end.
--
Emmanuel Lecharny
Symas.com
directory.apache.org