max size limit and paged searches

[Tobias Bocanegra <tr...@apache.org>]

Hi,

I have a simple test case [2] with a server where I set the
LdapServer#setMaxSizeLimit to 50 [0], and then do a paged search with
a page size of 10 [1]. the total number of objects in the search is
100, but I'm not able to get more than the max 50.

I thought that the paged search is exactly for that use case.

btw: executing the same code against a AD which has a limit of 1000
for simple searches, allows me to read all (~27k) entries.

regards, toby

[0] https://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-auth-ldap/src/test/java/org/apache/jackrabbit/oak/security/authentication/ldap/AbstractServer.java?revision=1689773&view=markup#l281
[1] https://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-auth-ldap/src/main/java/org/apache/jackrabbit/oak/security/authentication/ldap/impl/LdapIdentityProvider.java?revision=1689798&view=markup#l637
[2] https://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-auth-ldap/src/test/java/org/apache/jackrabbit/oak/security/authentication/ldap/LargeLdapProviderTest.java?revision=1689773&view=markup

Re: max size limit and paged searches

[Emmanuel Lécharny <el...@gmail.com>]

Le 08/07/15 17:45, Tobias Bocanegra a écrit :
> Hi Emmanuel,
>
> thanks for the explanation. So how is it then possible to read all the
> entries? 

Change the server's setting to have no size limit.

> Or is it just a configuration or structure problem of the
> server, if not all the entries can be read?
Yes.

> Should, in this case, the entries tree be more fragmented, in order to
> allow clients to traverse them all?

Again, the server sizeLimit is intended to protect the client from doing
stupid things (like fetching 10 000 000 entries from a server). Using
the pagedSearch control mitigate the problem. The server itself usually
handle without any problem a full scan.
>
> Reading all entries with (objectlclass=*) is of course not a real-life
> use case, but for example, reading all the groups where a user belongs
> to (if there are many groups :-)
The server will handle that easily. The PagedSearch control will help
not sending all the results.

Re: max size limit and paged searches

[Tobias Bocanegra <tr...@apache.org>]

Hi Emmanuel,

thanks for the explanation. So how is it then possible to read all the
entries? Or is it just a configuration or structure problem of the
server, if not all the entries can be read?
Should, in this case, the entries tree be more fragmented, in order to
allow clients to traverse them all?

Reading all entries with (objectlclass=*) is of course not a real-life
use case, but for example, reading all the groups where a user belongs
to (if there are many groups :-)

thanks.
regards, toby

On Wed, Jul 8, 2015 at 2:00 AM, Emmanuel Lécharny <el...@gmail.com> wrote:
> Hi Tobias,
>
> Le 08/07/15 08:33, Tobias Bocanegra a écrit :
>> Hi,
>>
>> I have a simple test case [2] with a server where I set the
>> LdapServer#setMaxSizeLimit to 50 [0], and then do a paged search with
>> a page size of 10 [1]. the total number of objects in the search is
>> 100, but I'm not able to get more than the max 50.
>>
>> I thought that the paged search is exactly for that use case.
> Well, no.
>
> The PagedSearch control (https://tools.ietf.org/html/rfc2696) says :
>
> "This control extension allows a client to control the rate at which an
> LDAP server returns the results of an LDAP search operation. This
> control may be useful when the LDAP client has limited resources and may
> not be able to process the entire result set from a given LDAP query, or
> when the LDAP client is connected over a low-bandwidth connection"
>
> As you can see, it's mainly to mitigate client side limitations, it's
> not supposed to overrule the server limits.
>
> If you look at OpenLDAP documentation, they have two limits on the
> server : a soft limit and a hard limit. The hard limit can't be
> overruled, except by the admin user. The soft limit is the one that is
> used when there is no limit set in the search request. In any case, you
> won't be able to fetch more than the server's hard size limit :
>
> http://www.openldap.org/doc/admin24/limits.html
>
> In ApacheDS, we don't have any soft limit, but we have a hard limit.
>
> AD has a different implementation, which allows you to read all the
> entries, whatever the server's sizeLimit is.
>

Re: max size limit and paged searches

[Emmanuel Lécharny <el...@gmail.com>]

Hi Tobias,

Le 08/07/15 08:33, Tobias Bocanegra a écrit :
> Hi,
>
> I have a simple test case [2] with a server where I set the
> LdapServer#setMaxSizeLimit to 50 [0], and then do a paged search with
> a page size of 10 [1]. the total number of objects in the search is
> 100, but I'm not able to get more than the max 50.
>
> I thought that the paged search is exactly for that use case.
Well, no.

The PagedSearch control (https://tools.ietf.org/html/rfc2696) says :

"This control extension allows a client to control the rate at which an
LDAP server returns the results of an LDAP search operation. This
control may be useful when the LDAP client has limited resources and may
not be able to process the entire result set from a given LDAP query, or
when the LDAP client is connected over a low-bandwidth connection"

As you can see, it's mainly to mitigate client side limitations, it's
not supposed to overrule the server limits.

If you look at OpenLDAP documentation, they have two limits on the
server : a soft limit and a hard limit. The hard limit can't be
overruled, except by the admin user. The soft limit is the one that is
used when there is no limit set in the search request. In any case, you
won't be able to fetch more than the server's hard size limit :

http://www.openldap.org/doc/admin24/limits.html

In ApacheDS, we don't have any soft limit, but we have a hard limit.

AD has a different implementation, which allows you to read all the
entries, whatever the server's sizeLimit is.