You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2013/11/24 10:51:41 UTC

[Mavibot Partition] Status

Hi guys,

this morning, I launched a test on core-integ to see where we are, after
a few modifications I have made on Mavibot :

Tests run: 690, Failures: 6, Errors: 12, Skipped: 132

This is better than last week, but still there is some progress.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Success

Posted by Kiran Ayyagari <ka...@apache.org>.
On Mon, Dec 2, 2013 at 12:48 AM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi guys,
>
> finally, I got all the tests passing ! I found a big big bug in Mavibot
> delete operation, which was leaving the BTree in a bad state (ie, some
> entries suposely being deleted was still present). Once I fixed this bug
> (it was easy), I had all the server-integ and all the other tests
> passing fine.
>
> I'm now pretty sure that this bug was there for quite a long time,
> because I had random failures before while running tests, and had to run
> the tests more than once to get them running. The random failure was
> linked to the fact that we were using UUID to index the master table,
> and as UUID were randomly generated made it possible that the tests were
> passing from time to time.
>
> In the process, I was able to speed up the add operation by removing the
> useless drop/add operation on the rdnIndex, which speed up the addition
> by 10%.
>

All in all, we have now a server which run faster than with JDBM.
>
> perfect, thanks for the heads up

> Although we are not ready yet to get a release done. There are a few
> failures in Mavibot (in-memory partition browse operation is somehow
> broken due to some modifications I have done 2 weeks ago) and there are
> some tuning to do in the configuration to add some support for the cache
> used in Mavibot. Bt this is nothng really big.
>
> I'm confident now that we have something decent, even if we have many
> things to add to have an even faster server :
> - we should add some cache for indexes
> - we don't yet support revisions in Mavibot (currently, we have quite
> the same thing than JDBM)
> - the cache is not efficient (ehCache is a bug ball of things, we dont
> need all of it)
> - searches performance get very low when we don't have enough memory,
> and this has to be fixed (JDBM behaves better in such case)
>
> Those are quite important improvements and enhancements to add to the
> server, but I think that could be done in a 2.0-RC2.
>
> We are not far at all from a RC1 !
>

> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com

[Mavibot Partition] Success

Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi guys,

finally, I got all the tests passing ! I found a big big bug in Mavibot
delete operation, which was leaving the BTree in a bad state (ie, some
entries suposely being deleted was still present). Once I fixed this bug
(it was easy), I had all the server-integ and all the other tests
passing fine.

I'm now pretty sure that this bug was there for quite a long time,
because I had random failures before while running tests, and had to run
the tests more than once to get them running. The random failure was
linked to the fact that we were using UUID to index the master table,
and as UUID were randomly generated made it possible that the tests were
passing from time to time.

In the process, I was able to speed up the add operation by removing the
useless drop/add operation on the rdnIndex, which speed up the addition
by 10%.

All in all, we have now a server which run faster than with JDBM.

Although we are not ready yet to get a release done. There are a few
failures in Mavibot (in-memory partition browse operation is somehow
broken due to some modifications I have done 2 weeks ago) and there are
some tuning to do in the configuration to add some support for the cache
used in Mavibot. Bt this is nothng really big.

I'm confident now that we have something decent, even if we have many
things to add to have an even faster server :
- we should add some cache for indexes
- we don't yet support revisions in Mavibot (currently, we have quite
the same thing than JDBM)
- the cache is not efficient (ehCache is a bug ball of things, we dont
need all of it)
- searches performance get very low when we don't have enough memory,
and this has to be fixed (JDBM behaves better in such case)

Those are quite important improvements and enhancements to add to the
server, but I think that could be done in a 2.0-RC2.

We are not far at all from a RC1 !

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Status

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/30/13 1:25 AM, Emmanuel Lécharny a écrit :
> Le 11/29/13 10:31 AM, Emmanuel Lécharny a écrit :
>> Le 11/28/13 7:26 PM, Emmanuel Lécharny a écrit :
>>> Le 11/24/13 12:50 PM, Emmanuel Lécharny a écrit :
>>>> Le 11/24/13 10:51 AM, Emmanuel Lécharny a écrit :
>>>>> Hi guys,
>>>>>
>>>>> this morning, I launched a test on core-integ to see where we are, after
>>>>> a few modifications I have made on Mavibot :
>>>>>
>>>>> Tests run: 690, Failures: 6, Errors: 12, Skipped: 132
>>>>>
>>>>> This is better than last week, but still there is some progress.
>>>>>
>>>> I forgot to mention that the failing tests are not failing when ran
>>>> individually. That mens there is more likely a problem with the Mavibot
>>>> partition shutdown and reload between two tests.
>>>>
>>> I was damn wrong...
>>>
>>> Many failures are due to bugs in mavibot, or bugs in the mavibot partition.
>>>
>>> I have fixed a serious issue in Mavibot itself, and I'm now chasing bugs
>>> in the mavibot partition. 6 errors and 6 failures in core-integ atm.
>>>
>> After a good insomnia, I was able to fix a couple of more bugs. We now
>> have :
>>
>> Tests run: 690, Failures: 3, Errors: 1, Skipped: 132
>>
>> The failing tests are :
>>
>> Failed tests:
>>   AliasSearchIT.testOuUsers:171->verifyCounts:557 expected:<2> but was:<1>
>>   AliasSearchIT.testOuPeople:194->verifyCounts:560 expected:<2> but was:<1>
>>   AliasSearchIT.testOuEngineering:274->verifyCounts:557 expected:<1> but
>> was:<0>
>>
>> Tests in error:
>>   PartitionConfigurationIT.testAddAndRemove:87 » LdapOperationError
>> ERR_577 Erro...
>>
>>
>> One of the problème was that we were'nt replacing an entry which content
>> was modified, leading to some errors when checking the attributes.
>>
>> I have to figure out what are those Alias issues. Will work on that this
>> evening or this week-end.
>>
> Ok, I now have core-integ tests passing ! The
> partitionConfigurationTestIT error was due to a double closure of the
> recordManager. This was easy to fix.
>
> I also modified the way we update the RdnIdex : as we now can replace an
> elemnt in a BTree, we don't anymore drop and add each ParentIdAndRdn
> into this index. The gain is 20% if speed when adding entries (140/s
> against 120/s previously).
>
> All the core-integ tests are now passing, I have failurs in server-integ :
>
> Failed tests:
>   SearchIT.testMultiValuedRdnContent:824 null
>   ClientInitialRefreshIT.testInitialRefreshLoad:360 null
>
> Tests in error:
>   SearchIT.testSearchJpeg:895 » Naming [LDAP: error code 80 - OTHER:
> failed for ...
>   SearchIT.testSearchUsersAttrs:1144 » Naming [LDAP: error code 80 -
> OTHER: fail...
>   SearchIT.testSearchAttrCommonName:1019 » Naming [LDAP: error code 80 -
> OTHER: ...
>   SearchIT.testSubstringSearchWithEscapedCharsInFilter:1484 » Naming
> [LDAP: erro...
>   SearchIT.testDirserver635:416->search:398 » Naming [LDAP: error code
> 80 - OTHE...
>   SearchIT.testSearchOperationalAttrs:1178 » Naming [LDAP: error code 80
> - OTHER...
>   SearchIT.testSearchAttrCN:958 » Naming [LDAP: error code 80 - OTHER:
> failed fo...
>   SearchIT.testSubentryControl:768->createAccessControlSubentry:685 »
> Naming [LD...
>   SearchIT.testSearchAttrName:987 » Naming [LDAP: error code 80 - OTHER:
> failed ...
>   SearchIT.testUndefinedAvaInBranchFilters:611->search:398 » Naming
> [LDAP: error...
>   SearchIT.testUnicodeFilter_DIRSERVER_1347:1601 » Naming [LDAP: error
> code 80 -...
>
>
> I will investigate them this week-end.
>
All of those test failures are due to one single test which left the
server in a bad state : the testAbandonnedRequest test is creating 100
users, and tries to abandon a search request done on those 100 users.
The test itself works, but we don't clean the server fast enough at the
end of the test (for some unknown reason).

If I @Ignore this test, all of the server-unit tests are passing.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Status

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/29/13 10:31 AM, Emmanuel Lécharny a écrit :
> Le 11/28/13 7:26 PM, Emmanuel Lécharny a écrit :
>> Le 11/24/13 12:50 PM, Emmanuel Lécharny a écrit :
>>> Le 11/24/13 10:51 AM, Emmanuel Lécharny a écrit :
>>>> Hi guys,
>>>>
>>>> this morning, I launched a test on core-integ to see where we are, after
>>>> a few modifications I have made on Mavibot :
>>>>
>>>> Tests run: 690, Failures: 6, Errors: 12, Skipped: 132
>>>>
>>>> This is better than last week, but still there is some progress.
>>>>
>>> I forgot to mention that the failing tests are not failing when ran
>>> individually. That mens there is more likely a problem with the Mavibot
>>> partition shutdown and reload between two tests.
>>>
>> I was damn wrong...
>>
>> Many failures are due to bugs in mavibot, or bugs in the mavibot partition.
>>
>> I have fixed a serious issue in Mavibot itself, and I'm now chasing bugs
>> in the mavibot partition. 6 errors and 6 failures in core-integ atm.
>>
> After a good insomnia, I was able to fix a couple of more bugs. We now
> have :
>
> Tests run: 690, Failures: 3, Errors: 1, Skipped: 132
>
> The failing tests are :
>
> Failed tests:
>   AliasSearchIT.testOuUsers:171->verifyCounts:557 expected:<2> but was:<1>
>   AliasSearchIT.testOuPeople:194->verifyCounts:560 expected:<2> but was:<1>
>   AliasSearchIT.testOuEngineering:274->verifyCounts:557 expected:<1> but
> was:<0>
>
> Tests in error:
>   PartitionConfigurationIT.testAddAndRemove:87 » LdapOperationError
> ERR_577 Erro...
>
>
> One of the problème was that we were'nt replacing an entry which content
> was modified, leading to some errors when checking the attributes.
>
> I have to figure out what are those Alias issues. Will work on that this
> evening or this week-end.
>
Ok, I now have core-integ tests passing ! The
partitionConfigurationTestIT error was due to a double closure of the
recordManager. This was easy to fix.

I also modified the way we update the RdnIdex : as we now can replace an
elemnt in a BTree, we don't anymore drop and add each ParentIdAndRdn
into this index. The gain is 20% if speed when adding entries (140/s
against 120/s previously).

All the core-integ tests are now passing, I have failurs in server-integ :

Failed tests:
  SearchIT.testMultiValuedRdnContent:824 null
  ClientInitialRefreshIT.testInitialRefreshLoad:360 null

Tests in error:
  SearchIT.testSearchJpeg:895 » Naming [LDAP: error code 80 - OTHER:
failed for ...
  SearchIT.testSearchUsersAttrs:1144 » Naming [LDAP: error code 80 -
OTHER: fail...
  SearchIT.testSearchAttrCommonName:1019 » Naming [LDAP: error code 80 -
OTHER: ...
  SearchIT.testSubstringSearchWithEscapedCharsInFilter:1484 » Naming
[LDAP: erro...
  SearchIT.testDirserver635:416->search:398 » Naming [LDAP: error code
80 - OTHE...
  SearchIT.testSearchOperationalAttrs:1178 » Naming [LDAP: error code 80
- OTHER...
  SearchIT.testSearchAttrCN:958 » Naming [LDAP: error code 80 - OTHER:
failed fo...
  SearchIT.testSubentryControl:768->createAccessControlSubentry:685 »
Naming [LD...
  SearchIT.testSearchAttrName:987 » Naming [LDAP: error code 80 - OTHER:
failed ...
  SearchIT.testUndefinedAvaInBranchFilters:611->search:398 » Naming
[LDAP: error...
  SearchIT.testUnicodeFilter_DIRSERVER_1347:1601 » Naming [LDAP: error
code 80 -...


I will investigate them this week-end.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Status

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/28/13 7:26 PM, Emmanuel Lécharny a écrit :
> Le 11/24/13 12:50 PM, Emmanuel Lécharny a écrit :
>> Le 11/24/13 10:51 AM, Emmanuel Lécharny a écrit :
>>> Hi guys,
>>>
>>> this morning, I launched a test on core-integ to see where we are, after
>>> a few modifications I have made on Mavibot :
>>>
>>> Tests run: 690, Failures: 6, Errors: 12, Skipped: 132
>>>
>>> This is better than last week, but still there is some progress.
>>>
>> I forgot to mention that the failing tests are not failing when ran
>> individually. That mens there is more likely a problem with the Mavibot
>> partition shutdown and reload between two tests.
>>
> I was damn wrong...
>
> Many failures are due to bugs in mavibot, or bugs in the mavibot partition.
>
> I have fixed a serious issue in Mavibot itself, and I'm now chasing bugs
> in the mavibot partition. 6 errors and 6 failures in core-integ atm.
>
After a good insomnia, I was able to fix a couple of more bugs. We now
have :

Tests run: 690, Failures: 3, Errors: 1, Skipped: 132

The failing tests are :

Failed tests:
  AliasSearchIT.testOuUsers:171->verifyCounts:557 expected:<2> but was:<1>
  AliasSearchIT.testOuPeople:194->verifyCounts:560 expected:<2> but was:<1>
  AliasSearchIT.testOuEngineering:274->verifyCounts:557 expected:<1> but
was:<0>

Tests in error:
  PartitionConfigurationIT.testAddAndRemove:87 » LdapOperationError
ERR_577 Erro...


One of the problème was that we were'nt replacing an entry which content
was modified, leading to some errors when checking the attributes.

I have to figure out what are those Alias issues. Will work on that this
evening or this week-end.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Status

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/24/13 12:50 PM, Emmanuel Lécharny a écrit :
> Le 11/24/13 10:51 AM, Emmanuel Lécharny a écrit :
>> Hi guys,
>>
>> this morning, I launched a test on core-integ to see where we are, after
>> a few modifications I have made on Mavibot :
>>
>> Tests run: 690, Failures: 6, Errors: 12, Skipped: 132
>>
>> This is better than last week, but still there is some progress.
>>
> I forgot to mention that the failing tests are not failing when ran
> individually. That mens there is more likely a problem with the Mavibot
> partition shutdown and reload between two tests.
>
I was damn wrong...

Many failures are due to bugs in mavibot, or bugs in the mavibot partition.

I have fixed a serious issue in Mavibot itself, and I'm now chasing bugs
in the mavibot partition. 6 errors and 6 failures in core-integ atm.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 


Re: [Mavibot Partition] Status

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/24/13 10:51 AM, Emmanuel Lécharny a écrit :
> Hi guys,
>
> this morning, I launched a test on core-integ to see where we are, after
> a few modifications I have made on Mavibot :
>
> Tests run: 690, Failures: 6, Errors: 12, Skipped: 132
>
> This is better than last week, but still there is some progress.
>
I forgot to mention that the failing tests are not failing when ran
individually. That mens there is more likely a problem with the Mavibot
partition shutdown and reload between two tests.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com