You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2013/11/10 10:25:57 UTC
Ads + Mavibot Partition
Just quick heads up :
Im tryinh to fix a bug that forbid us to inject a few thousands entries
in the server (the injection test fails after aound 3000 thousands of
entries added). The problem lies in teh way we serialize/deserialize
ParentIdAndRdn. For some reason, we add the length of the byte[]
containing the ParentIdAndRdn before the serialized value. The
deserializer does not expect to find this length, thought the next
position is not good. If I remove this length, now I have a pb because
the number of expected RDNs is too big, and I get an
ArrayOutofCoundException while tryingt to create an array of RDN[] with
zillions of values.
Obviously, we don't serialize/deserialize correctly, although the
ser/deser methods have been tested and work. It's more about the way we
call them : when we reach a number of stored elements, the cache is
full, and we need to read back some pages from disk. This is when we
have the issue. I have to understand why a Page deserialization fails.
It should not be such a big issue, but the failure is quite random
(something I don't understand, though... Not sure that the cache is not
responsible for this randomness).
I'll keep the list updated.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Ads + Mavibot Partition
Posted by Emmanuel Lécharny <el...@gmail.com>.
Ok, pb fixed.
It was slightly more complex that I expected. The key length wasn't
stored for nodes, as shown in my previous mail, but the impact was a bit
more than just what I thought (ie, the fix was not what I suggested).
Le 11/11/13 8:14 AM, Emmanuel Lécharny a écrit :
> Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
>> Just quick heads up :
>>
>> Im tryinh to fix a bug that forbid us to inject a few thousands entries
>> in the server (the injection test fails after aound 3000 thousands of
>> entries added). The problem lies in teh way we serialize/deserialize
>> ParentIdAndRdn. For some reason, we add the length of the byte[]
>> containing the ParentIdAndRdn before the serialized value. The
>> deserializer does not expect to find this length, thought the next
>> position is not good. If I remove this length, now I have a pb because
>> the number of expected RDNs is too big, and I get an
>> ArrayOutofCoundException while tryingt to create an array of RDN[] with
>> zillions of values.
>>
>> Obviously, we don't serialize/deserialize correctly, although the
>> ser/deser methods have been tested and work. It's more about the way we
>> call them : when we reach a number of stored elements, the cache is
>> full, and we need to read back some pages from disk. This is when we
>> have the issue. I have to understand why a Page deserialization fails.
>> It should not be such a big issue, but the failure is quite random
>> (something I don't understand, though... Not sure that the cache is not
>> responsible for this randomness).
>>
>> I'll keep the list updated.
>>
> Ok, found the issue.
>
> We have to store the length of a serialized key before the serialized
> key, otherwise we have no way to know where to start the deserialization
> of the next value. The code that deserialize a page is like :
>
> // Read each value and key
> for ( int i = 0; i < nbElems; i++ )
> {
> // Read the value (page offsets)
> long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
> long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );
>
> ...
>
> // And the key
> K key = btree.getKeySerializer().deserialize( byteBuffer );
> }
>
> Here, we deserialize the key passing a ByteBuffer, though the position
> in this ByteBuffer should be changed after the deserialization.
>
> This is correctly done in the MavibotParentIdAndRdnSerializer :
>
> @Override
> public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
> IOException
> {
> int len = buffer.getInt();
>
> ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
> buffer.position() );
>
> buffer.position( buffer.position() + len );
>
> return parentIdAndRdn;
> }
>
> Although in the same class, the serialization does not correctly set
> this value :
>
> public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
> {
> try
> {
> int bufferSize = 1024;
>
> while ( bufferSize < Integer.MAX_VALUE )
> {
> // allocate a big enough buffer for most of the cases
> byte[] buffer = new byte[bufferSize];
>
> try
> {
> // The current position. Start at 4, as we will add
> the length at pos 0 <<<----------------- Here we are missing something
> int pos = 0;
> <----------------------------------------------------------------------------
> It should be 4
>
> ...
>
> byte[] result = new byte[pos];
> <------------------------------------------------------------- We don't
> store the byte[] length
> System.arraycopy( buffer, 0, result, 0, pos );
>
> return result;
>
> So we just have to fix that.
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Ads + Mavibot Partition
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/10/13 10:25 AM, Emmanuel Lécharny a écrit :
> Just quick heads up :
>
> Im tryinh to fix a bug that forbid us to inject a few thousands entries
> in the server (the injection test fails after aound 3000 thousands of
> entries added). The problem lies in teh way we serialize/deserialize
> ParentIdAndRdn. For some reason, we add the length of the byte[]
> containing the ParentIdAndRdn before the serialized value. The
> deserializer does not expect to find this length, thought the next
> position is not good. If I remove this length, now I have a pb because
> the number of expected RDNs is too big, and I get an
> ArrayOutofCoundException while tryingt to create an array of RDN[] with
> zillions of values.
>
> Obviously, we don't serialize/deserialize correctly, although the
> ser/deser methods have been tested and work. It's more about the way we
> call them : when we reach a number of stored elements, the cache is
> full, and we need to read back some pages from disk. This is when we
> have the issue. I have to understand why a Page deserialization fails.
> It should not be such a big issue, but the failure is quite random
> (something I don't understand, though... Not sure that the cache is not
> responsible for this randomness).
>
> I'll keep the list updated.
>
Ok, found the issue.
We have to store the length of a serialized key before the serialized
key, otherwise we have no way to know where to start the deserialization
of the next value. The code that deserialize a page is like :
// Read each value and key
for ( int i = 0; i < nbElems; i++ )
{
// Read the value (page offsets)
long offset = OFFSET_SERIALIZER.deserialize( byteBuffer );
long lastOffset = OFFSET_SERIALIZER.deserialize( byteBuffer );
...
// And the key
K key = btree.getKeySerializer().deserialize( byteBuffer );
}
Here, we deserialize the key passing a ByteBuffer, though the position
in this ByteBuffer should be changed after the deserialization.
This is correctly done in the MavibotParentIdAndRdnSerializer :
@Override
public ParentIdAndRdn deserialize( ByteBuffer buffer ) throws
IOException
{
int len = buffer.getInt();
ParentIdAndRdn parentIdAndRdn = fromBytes( buffer.array(),
buffer.position() );
buffer.position( buffer.position() + len );
return parentIdAndRdn;
}
Although in the same class, the serialization does not correctly set
this value :
public byte[] serialize( ParentIdAndRdn parentIdAndRdn )
{
try
{
int bufferSize = 1024;
while ( bufferSize < Integer.MAX_VALUE )
{
// allocate a big enough buffer for most of the cases
byte[] buffer = new byte[bufferSize];
try
{
// The current position. Start at 4, as we will add
the length at pos 0 <<<----------------- Here we are missing something
int pos = 0;
<----------------------------------------------------------------------------
It should be 4
...
byte[] result = new byte[pos];
<------------------------------------------------------------- We don't
store the byte[] length
System.arraycopy( buffer, 0, result, 0, pos );
return result;
So we just have to fix that.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com