You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lecharny <el...@gmail.com> on 2010/05/17 01:48:21 UTC
Paged Search issues
Hi guys,
as everyone who build the server from time to time knows, we have random
errors in PagedSearch tests.
Stefan spent some time to see what's going on, and I must admit I was
lazzy enough to accept the idea that we have errors in this area, until
I move my butt to fix it.
So I checked the code, and I found a couple of major errors in this code
(and as I wrote it, I'm responsible for this) :
- first, I wrongly assumed that the cookie would be 4 bytes long, as it
was stored in an int on the server. Nothing could be more wrong, as an
int BER encoded can be from 1 to 4 bytes. We had situation where, when
the cookie was encoded on 3 bytes only, we had an internal AIOOBE. I
fixed that.
- second, the cookie was computed based on the message id plus a time
based number (System.nanoseconds(), namely). This is *stupid*. There are
potential collisions when computig such a cookie. I changed it to use an
AtomicInteger - which as a side effect removed the local synchronization
- and a 16 bits shift to the left to allow 2^16 cookie renewig for a
single message - more than enough to cover every cases.
I still have random issues at this point, but less frequently. What
happens is that we receive a searchRequest with a cookie, just after the
previous request has been completed. I have no idea why. If I add a
Thread.sleep(10) in the doLoop() method of the test, I'm able to run
10000 times the paged search ( this is 40 000 paged search requests...).
I will try tomorrow the same test but using our API instead of JNDI.
Will keep you informed.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Paged Search issues
Posted by Stefan Seelmann <se...@apache.org>.
Thanks Emmanuel for handling this.
Emmanuel Lecharny wrote:
> Hi guys,
>
> as everyone who build the server from time to time knows, we have random
> errors in PagedSearch tests.
>
> Stefan spent some time to see what's going on, and I must admit I was
> lazzy enough to accept the idea that we have errors in this area, until
> I move my butt to fix it.
>
> So I checked the code, and I found a couple of major errors in this code
> (and as I wrote it, I'm responsible for this) :
> - first, I wrongly assumed that the cookie would be 4 bytes long, as it
> was stored in an int on the server. Nothing could be more wrong, as an
> int BER encoded can be from 1 to 4 bytes. We had situation where, when
> the cookie was encoded on 3 bytes only, we had an internal AIOOBE. I
> fixed that.
> - second, the cookie was computed based on the message id plus a time
> based number (System.nanoseconds(), namely). This is *stupid*. There are
> potential collisions when computig such a cookie. I changed it to use an
> AtomicInteger - which as a side effect removed the local synchronization
> - and a 16 bits shift to the left to allow 2^16 cookie renewig for a
> single message - more than enough to cover every cases.
>
> I still have random issues at this point, but less frequently. What
> happens is that we receive a searchRequest with a cookie, just after the
> previous request has been completed. I have no idea why. If I add a
> Thread.sleep(10) in the doLoop() method of the test, I'm able to run
> 10000 times the paged search ( this is 40 000 paged search requests...).
>
> I will try tomorrow the same test but using our API instead of JNDI.
>
> Will keep you informed.
>
Re: Paged Search issues
Posted by Emmanuel Lecharny <el...@gmail.com>.
Two things :
- I just committed some fix on the client side : the connection is now
closed for every test, and I added a delay (10ms) to be sure that the
Unbind request is correctly send before the new Bind request - it seems
to be the problem we have -
- Regarding the getCookie() returning a reference, this is a case where
the cookie is *never* modified on the server.
The PagedSearch tests are now passing well on my computer, I have loop
100 000 times on the first test, without any issue.
I would suggest that *if* we are to add @SuppressWarning all over the
code, then a comment must be added explaining why we are suppressing a
warning. Otherwise, we may have bad surprises in the near future. (it's
like pushing dusts under the rug...)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Paged Search issues
Posted by Emmanuel Lecharny <el...@gmail.com>.
Why the fuck do we receive such PDU ???
0x30 0x48
0x02 0x01 0x06
0x42 0x00 <--- This is an Unbind request
0xA0 0x41 <-- This is a bloody PagedSearchControl, wtf is it doing
here ???
0x30 0x24
0x04 0x16
0x31 0x2E 0x32 0x2E 0x38 0x34 0x30 0x2E
0x31 0x31 0x33 0x35 0x35 0x36 0x2E 0x31
0x2E 0x34 0x2E 0x33 0x31 0x39
0x04 0x0A
0x30 0x08
0x02 0x01 0x03
0x04 0x03
0x02 0x00 0x00
0x30 0x19
0x04 0x17
0x32 0x2E 0x31 0x36 0x2E 0x38 0x34 0x30
0x2E 0x31 0x2E 0x31 0x31 0x33 0x37 0x33
0x30 0x2E 0x33 0x2E 0x34 0x2E 0x32
It seems that JNDI does not know how to correctly send an Unbind request...
Also I've dumped the received PDU and when we are hit by the problem (ie
LdapError 53 : bad cookie received), it shodws that we don't receive the
UnbindRequest fast enough, thus we get a BindRequest immediately
followed by a SearchRequest containing a cookie which is not reset. As
the LdapSession has been reset by the new Bind, the transmitted cookie
is obviously not present in this session, and we get the error we are
facing.
So far, it's absolutely normal, and I really suspect that this is a JNDI
problem. I have added an explicitely context close(), with no help.
My last test was to add a 10 ms wait after the ctx.close() in the
doLoop() method and it seems to be efficient.
I will try with the Ldap API to see if we are havig the same problem.
On 5/17/10 9:27 AM, Felix Knecht wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Good catch Emmanuel.
> Maybe this [1] could be a reason (just from far away without any deeper
> knowledge).
>
> [1]
> http://people.apache.org/~felixk/apacheds-docs/apacheds-protocol-ldap/findbugs.html#org.apache.directory.server.ldap.handlers.controls.PagedSearchContext
>
>
>> I still have random issues at this point, but less frequently. What
>> happens is that we receive a searchRequest with a cookie, just after the
>> previous request has been completed. I have no idea why. If I add a
>> Thread.sleep(10) in the doLoop() method of the test, I'm able to run
>> 10000 times the paged search ( this is 40 000 paged search requests...).
>>
>> I will try tomorrow the same test but using our API instead of JNDI.
>>
>> Will keep you informed.
>>
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.15 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkvw7/UACgkQ2lZVCB08qHFCLwCg4yTi8jF4HO+8wfVQKPet+dvg
> VFgAoKlZvIZ9ozf/2wlKi1DRENEQwm9C
> =2PAe
> -----END PGP SIGNATURE-----
>
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Paged Search issues
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/17/10 9:27 AM, Felix Knecht wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Good catch Emmanuel.
> Maybe this [1] could be a reason (just from far away without any deeper
> knowledge).
>
You now what ? I gonna copy the cookie instead of returning the
reference. I don't know if it will fix the issue, but if it does, it
will be a quicker solution than any other !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
Re: Paged Search issues
Posted by Felix Knecht <fe...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Good catch Emmanuel.
Maybe this [1] could be a reason (just from far away without any deeper
knowledge).
[1]
http://people.apache.org/~felixk/apacheds-docs/apacheds-protocol-ldap/findbugs.html#org.apache.directory.server.ldap.handlers.controls.PagedSearchContext
> I still have random issues at this point, but less frequently. What
> happens is that we receive a searchRequest with a cookie, just after the
> previous request has been completed. I have no idea why. If I add a
> Thread.sleep(10) in the doLoop() method of the test, I'm able to run
> 10000 times the paged search ( this is 40 000 paged search requests...).
>
> I will try tomorrow the same test but using our API instead of JNDI.
>
> Will keep you informed.
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkvw7/UACgkQ2lZVCB08qHFCLwCg4yTi8jF4HO+8wfVQKPet+dvg
VFgAoKlZvIZ9ozf/2wlKi1DRENEQwm9C
=2PAe
-----END PGP SIGNATURE-----