You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Stefan Seelmann <ma...@stefan-seelmann.de> on 2013/08/10 15:08:32 UTC
[Mavibot] Journal in BTree and RecordManager
Hi,
I have a question regarding the journal. The BTree class creates a
journal in case the type is PERSISTENT. But if the BTree is managed by
the record manager then there is no journal. Is that intended, or is it
not yet implemented?
Kind Regards,
Stefan
Re: [Mavibot] Journal in BTree and RecordManager
Posted by Kiran Ayyagari <ka...@apache.org>.
On Sun, Aug 11, 2013 at 1:07 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 8/10/13 5:19 PM, Kiran Ayyagari a écrit :
> > On Sat, Aug 10, 2013 at 8:30 PM, Stefan Seelmann <
> mail@stefan-seelmann.de>wrote:
> >
> >> Thanks for the response Kiran.
> >>
> >> In the meantime I think I found some problems in the BTree journal
> >> manager thread and the BTreeFlushTest.
> >>
> >> 1. On Jenkins Windows Build the BTreeFlushTest failed [1]. The failing
> >> assertion checks if the length of the journal file is greater than 0. As
> >> the journal manager runs in its own thread it is not guaranteed that the
> >> thread run and the journal may not be written. I was able to reproduce
> >> it also on Linux when using the MultiThreadedMultiInvoker. I wonder why
> >> the journal uses its own thread at all, I thought each modification
> >> should immediately written and flushed to the journal?
> >>
> > yes, agree, not sure what was the reason behind it, Emmanuel may have a
> > better answer to this
>
> Well, I decided to accept the idea that we may not save all the
> modifications in case of a crash, but as much as we can. The
> modifications are pushed into a blocking queue, and the journal thread
> process them as soon as they are some entries in this queue. Doing so
> allow the main thread to write fast.
>
> t may bnot be the best solution.
>
fixed as per MAVIBOT-5
> >
> >> 2. I think there is a bug in the BTree's journal manager code. The
> >> flushModification() method returns a boolean. It is true when a Addition
> >> or a Deletion was added, but then the journal manager thread stops. I
> >> guess the boolean value should be inverted?
> >>
> >> yeap, it is a bug, the return values need to be inverted
>
> +1
>
these flags are no longer needed, removed them as well as the journal
thread
> > Emmanuel,
> >
> > I think the journal be deleted every time we flush(by calling flush())
> > the data to disk?
> > cause at line 245 we have the below assertion, but journal has data
> > cause we are not
> > truncating it (I ran the testFlushBTree() after inverting the above
> said
> > return values)
> > // The journal must be empty
> > assertEquals( 0, journal.length() );
>
> When we flush the btree, the journal should be emptied.
>
> done, the journal gets truncated using its channel
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
--
Kiran Ayyagari
http://keydap.com
Re: [Mavibot] Journal in BTree and RecordManager
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 8/10/13 5:19 PM, Kiran Ayyagari a écrit :
> On Sat, Aug 10, 2013 at 8:30 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
>
>> Thanks for the response Kiran.
>>
>> In the meantime I think I found some problems in the BTree journal
>> manager thread and the BTreeFlushTest.
>>
>> 1. On Jenkins Windows Build the BTreeFlushTest failed [1]. The failing
>> assertion checks if the length of the journal file is greater than 0. As
>> the journal manager runs in its own thread it is not guaranteed that the
>> thread run and the journal may not be written. I was able to reproduce
>> it also on Linux when using the MultiThreadedMultiInvoker. I wonder why
>> the journal uses its own thread at all, I thought each modification
>> should immediately written and flushed to the journal?
>>
> yes, agree, not sure what was the reason behind it, Emmanuel may have a
> better answer to this
Well, I decided to accept the idea that we may not save all the
modifications in case of a crash, but as much as we can. The
modifications are pushed into a blocking queue, and the journal thread
process them as soon as they are some entries in this queue. Doing so
allow the main thread to write fast.
t may bnot be the best solution.
>
>> 2. I think there is a bug in the BTree's journal manager code. The
>> flushModification() method returns a boolean. It is true when a Addition
>> or a Deletion was added, but then the journal manager thread stops. I
>> guess the boolean value should be inverted?
>>
>> yeap, it is a bug, the return values need to be inverted
+1
> Emmanuel,
>
> I think the journal be deleted every time we flush(by calling flush())
> the data to disk?
> cause at line 245 we have the below assertion, but journal has data
> cause we are not
> truncating it (I ran the testFlushBTree() after inverting the above said
> return values)
> // The journal must be empty
> assertEquals( 0, journal.length() );
When we flush the btree, the journal should be emptied.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: [Mavibot] Journal in BTree and RecordManager
Posted by Kiran Ayyagari <ka...@apache.org>.
On Sat, Aug 10, 2013 at 8:30 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
> Thanks for the response Kiran.
>
> In the meantime I think I found some problems in the BTree journal
> manager thread and the BTreeFlushTest.
>
> 1. On Jenkins Windows Build the BTreeFlushTest failed [1]. The failing
> assertion checks if the length of the journal file is greater than 0. As
> the journal manager runs in its own thread it is not guaranteed that the
> thread run and the journal may not be written. I was able to reproduce
> it also on Linux when using the MultiThreadedMultiInvoker. I wonder why
> the journal uses its own thread at all, I thought each modification
> should immediately written and flushed to the journal?
>
yes, agree, not sure what was the reason behind it, Emmanuel may have a
better answer to this
>
> 2. I think there is a bug in the BTree's journal manager code. The
> flushModification() method returns a boolean. It is true when a Addition
> or a Deletion was added, but then the journal manager thread stops. I
> guess the boolean value should be inverted?
>
> yeap, it is a bug, the return values need to be inverted
Emmanuel,
I think the journal be deleted every time we flush(by calling flush())
the data to disk?
cause at line 245 we have the below assertion, but journal has data
cause we are not
truncating it (I ran the testFlushBTree() after inverting the above said
return values)
// The journal must be empty
assertEquals( 0, journal.length() );
Kind Regards,
> Stefan
>
>
> [1]
>
> https://builds.apache.org/view/A-D/view/Directory/job/dir-mavibot-win/org.apache.directory.mavibot$mavibot/6/testReport/junit/org.apache.directory.mavibot.btree/BTreeFlushTest/testFlushBTree/
>
>
> On 08/10/2013 04:03 PM, Kiran Ayyagari wrote:
> > Hi Stefan,
> >
> >
> > On Sat, Aug 10, 2013 at 6:38 PM, Stefan Seelmann <
> mail@stefan-seelmann.de>wrote:
> >
> >> Hi,
> >>
> >> I have a question regarding the journal. The BTree class creates a
> >> journal in case the type is PERSISTENT. But if the BTree is managed by
> >> the record manager then there is no journal. Is that intended, or is it
> >> not yet implemented?
> >>
> >> yes, in managed mode there will be a different file and it contains more
> > than
> > one BTree, persist mode is intended to flush the contents of a single
> tree
> > to
> > a single file (i.e., the journal)
> >
> > the binary file format is different in both cases
> >
> >> Kind Regards,
> >> Stefan
> >>
> >
> >
> >
>
>
--
Kiran Ayyagari
http://keydap.com
Re: [Mavibot] Journal in BTree and RecordManager
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Thanks for the response Kiran.
In the meantime I think I found some problems in the BTree journal
manager thread and the BTreeFlushTest.
1. On Jenkins Windows Build the BTreeFlushTest failed [1]. The failing
assertion checks if the length of the journal file is greater than 0. As
the journal manager runs in its own thread it is not guaranteed that the
thread run and the journal may not be written. I was able to reproduce
it also on Linux when using the MultiThreadedMultiInvoker. I wonder why
the journal uses its own thread at all, I thought each modification
should immediately written and flushed to the journal?
2. I think there is a bug in the BTree's journal manager code. The
flushModification() method returns a boolean. It is true when a Addition
or a Deletion was added, but then the journal manager thread stops. I
guess the boolean value should be inverted?
Kind Regards,
Stefan
[1]
https://builds.apache.org/view/A-D/view/Directory/job/dir-mavibot-win/org.apache.directory.mavibot$mavibot/6/testReport/junit/org.apache.directory.mavibot.btree/BTreeFlushTest/testFlushBTree/
On 08/10/2013 04:03 PM, Kiran Ayyagari wrote:
> Hi Stefan,
>
>
> On Sat, Aug 10, 2013 at 6:38 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
>
>> Hi,
>>
>> I have a question regarding the journal. The BTree class creates a
>> journal in case the type is PERSISTENT. But if the BTree is managed by
>> the record manager then there is no journal. Is that intended, or is it
>> not yet implemented?
>>
>> yes, in managed mode there will be a different file and it contains more
> than
> one BTree, persist mode is intended to flush the contents of a single tree
> to
> a single file (i.e., the journal)
>
> the binary file format is different in both cases
>
>> Kind Regards,
>> Stefan
>>
>
>
>
Re: [Mavibot] Journal in BTree and RecordManager
Posted by Kiran Ayyagari <ka...@apache.org>.
Hi Stefan,
On Sat, Aug 10, 2013 at 6:38 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
> Hi,
>
> I have a question regarding the journal. The BTree class creates a
> journal in case the type is PERSISTENT. But if the BTree is managed by
> the record manager then there is no journal. Is that intended, or is it
> not yet implemented?
>
> yes, in managed mode there will be a different file and it contains more
than
one BTree, persist mode is intended to flush the contents of a single tree
to
a single file (i.e., the journal)
the binary file format is different in both cases
> Kind Regards,
> Stefan
>
--
Kiran Ayyagari
http://keydap.com