You are viewing a plain text version of this content. The canonical link for it is here.
Posted to api@directory.apache.org by "Harris, Christopher P" <ch...@baxter.com> on 2015/01/27 06:11:09 UTC
Proper use of LdapConnectionPool
Hi,
I'm running into TimeOut issues when implementing a multi-threaded approach to read all of LDAP/AD into memory, starting with the CEO and trickling down. I haven't been using the LdapConnectionPool, so I thought that I'd give that a try to see if it solves my TimeOut issues. I'm setting the timeout for my cursor and connection to 300,000 milliseconds, however the timeout consistently occurs around 30 seconds I've noticed.
Anyway, I read the docs concerning implementing an LdapConnectionPool, but the example doesn't show you how to use the connection pool.
Does the following method use the LdapConnectionPool the correct way?
public Person searchLdapUsingConnectionPool() {
SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
LdapConnectionPool pool = null;
LdapConnection connection = null;
Person p = null;
try {
LdapConnectionConfig config = new LdapConnectionConfig();
config.setLdapHost( host );
config.setLdapPort( port );
config.setName( dn );
config.setCredentials( pwd );
DefaultPoolableLdapConnectionFactory factory = new DefaultPoolableLdapConnectionFactory( config );
pool = new LdapConnectionPool( factory );
pool.setTestOnBorrow( true );
connection = pool.getConnection();
Entry entry = null;
SearchRequest sr = new SearchRequestImpl();
sr.setBase(new Dn(searchBase));
StringBuilder sb = new StringBuilder(ceoQuery);
sr.setFilter(sb.toString());
sr.setScope( SearchScope.SUBTREE );
cursor = connection.search(sr);
Response response;
while (cursor.next() && cursor.isEntry()) {
response = cursor.get();
System.out.println(((SearchResultEntry)response).getEntry());
entry = cursor.getEntry();
EntryMapper<Person> em = Person.getEntryMapper();
p = em.map(entry);
}
} catch (LdapException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} catch (CursorException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} finally {
cursor.close();
try {
pool.releaseConnection(connection);
} catch (LdapException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
}
return p;
}
It seems like I just need to grab a connection via pool.getConnection(), but I don't know for sure if I'm using it the recommended way. The method is working as expected and returning a Person object.
- Chris
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 04/02/15 00:11, Harris, Christopher P a écrit :
> Stefan,
>
> The hierarchy and logical organization are indeed different.
>
> However, even if they weren't different, I'd still be left with the problem that OBJECT or ONE_LEVEL don't seem to work and SUBTREE seems to act like OBJECT for this particular AD instance.
There *must* be somtehing configured in your AD server that forbid you
to do that. You must find out what it is, and fix it. Doing a OBJECT
(kind of) search on tens of thousands entry is just insanely slow.
Sadly, I've checked on the internet for a description of AD config, but
can't find anything related. Anyway, what would you expect from Microsoft...
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Stefan,
The hierarchy and logical organization are indeed different.
However, even if they weren't different, I'd still be left with the problem that OBJECT or ONE_LEVEL don't seem to work and SUBTREE seems to act like OBJECT for this particular AD instance.
- Chris
-----Original Message-----
From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
Sent: Tuesday, February 03, 2015 3:49 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
On 02/03/2015 10:19 PM, Emmanuel Lécharny wrote:
> Le 03/02/15 22:07, Stefan Seelmann a écrit :
>> I forgot to mention the performance aspect.
>>
>> If you traverse all persons from the CEO down you need as many LDAP
>> search operations as you have persons in the directory, each require
>> a full network roundtrip, which takes time.
>
> What's the point of doing that when a ONE_LEVEL search done one level
> below would provide all the entries with one single Search ?
If I understand Chris correctly the directory hierarchy and the logical organisational hierarchy are different. For example:
dn: cn=ceo,ou=c,ou=b,ou=a
directreports: cn=jane,ou=x,ou=w,ou=a
directreports: cn=john,ou=z,ou=y,ou=a
If that is the case the "directreports" are not LDAP child entries, but just pointer to somewhere in the directory tree. Similar to nested group membership.
@Chris, if this is not the case then please ignore my mail and use ONE_LEVEL search to traverse the directory hierarchy, that is the preferred way.
>> However if possible I'd avoid such a costly tree traversal, and
>> instead use e.g. paged search.
> Actually, it's probably better to abandon the request when you get
> what you want, paged search will just mitigate the memory used on the
> client side.
>
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 04/02/15 00:26, Harris, Christopher P a écrit :
> Emmanuel/Stefan,
>
> Yes. I'm currently exploring this route. It seems like the easier approach.
>
> The only issue that I'm encountering now is that the AD admins have set a cap on the number of returned entries to 20,000.
That a limit you can change on AD, but you need to be an administrator
of this server.
Another option is to use PagedSearch in this case, allowing you to get
data by chuncks, as suggested by Stefan.
>
> Our AD tree is primarily organized into global regions. So, I can set each region as a search base, construct a search filter that utilizes many criteria, and execute a search per region. Some regions will have to be broken into sub-regions, because some regions, such as North America and Europe contain over 20,000 entries so far...
>
> Still, I may need to employ multi-threading for each AD search against a region to speed things along.
As soon as your client is fast enough to deal with it, I see no reason
not to use one thread per region.
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Emmanuel/Stefan,
Yes. I'm currently exploring this route. It seems like the easier approach.
The only issue that I'm encountering now is that the AD admins have set a cap on the number of returned entries to 20,000.
Our AD tree is primarily organized into global regions. So, I can set each region as a search base, construct a search filter that utilizes many criteria, and execute a search per region. Some regions will have to be broken into sub-regions, because some regions, such as North America and Europe contain over 20,000 entries so far...
Still, I may need to employ multi-threading for each AD search against a region to speed things along. This entry compilation process will need to be quick, so I may get to employ some of your suggestions, Stefan.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, February 03, 2015 5:09 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 03/02/15 22:48, Stefan Seelmann a écrit :
> On 02/03/2015 10:19 PM, Emmanuel Lécharny wrote:
>> Le 03/02/15 22:07, Stefan Seelmann a écrit :
>>> I forgot to mention the performance aspect.
>>>
>>> If you traverse all persons from the CEO down you need as many LDAP
>>> search operations as you have persons in the directory, each require
>>> a full network roundtrip, which takes time.
>> What's the point of doing that when a ONE_LEVEL search done one level
>> below would provide all the entries with one single Search ?
> If I understand Chris correctly the directory hierarchy and the
> logical organisational hierarchy are different. For example:
>
> dn: cn=ceo,ou=c,ou=b,ou=a
> directreports: cn=jane,ou=x,ou=w,ou=a
> directreports: cn=john,ou=z,ou=y,ou=a
>
> If that is the case the "directreports" are not LDAP child entries,
> but just pointer to somewhere in the directory tree. Similar to nested
> group membership.
I see.
In this case, I would use a SubTree, and ditch entries that don't match the selection criteria. I think that it would be faster than creating new connections on the flight, despite the outrageaous number of entries transmoited. To be tested...
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 03/02/15 22:48, Stefan Seelmann a écrit :
> On 02/03/2015 10:19 PM, Emmanuel Lécharny wrote:
>> Le 03/02/15 22:07, Stefan Seelmann a écrit :
>>> I forgot to mention the performance aspect.
>>>
>>> If you traverse all persons from the CEO down you need as many LDAP
>>> search operations as you have persons in the directory, each require a
>>> full network roundtrip, which takes time.
>> What's the point of doing that when a ONE_LEVEL search done one level
>> below would provide all the entries with one single Search ?
> If I understand Chris correctly the directory hierarchy and the logical
> organisational hierarchy are different. For example:
>
> dn: cn=ceo,ou=c,ou=b,ou=a
> directreports: cn=jane,ou=x,ou=w,ou=a
> directreports: cn=john,ou=z,ou=y,ou=a
>
> If that is the case the "directreports" are not LDAP child entries, but
> just pointer to somewhere in the directory tree. Similar to nested group
> membership.
I see.
In this case, I would use a SubTree, and ditch entries that don't match
the selection criteria. I think that it would be faster than creating
new connections on the flight, despite the outrageaous number of entries
transmoited. To be tested...
Re: Proper use of LdapConnectionPool
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 02/03/2015 10:19 PM, Emmanuel Lécharny wrote:
> Le 03/02/15 22:07, Stefan Seelmann a écrit :
>> I forgot to mention the performance aspect.
>>
>> If you traverse all persons from the CEO down you need as many LDAP
>> search operations as you have persons in the directory, each require a
>> full network roundtrip, which takes time.
>
> What's the point of doing that when a ONE_LEVEL search done one level
> below would provide all the entries with one single Search ?
If I understand Chris correctly the directory hierarchy and the logical
organisational hierarchy are different. For example:
dn: cn=ceo,ou=c,ou=b,ou=a
directreports: cn=jane,ou=x,ou=w,ou=a
directreports: cn=john,ou=z,ou=y,ou=a
If that is the case the "directreports" are not LDAP child entries, but
just pointer to somewhere in the directory tree. Similar to nested group
membership.
@Chris, if this is not the case then please ignore my mail and use
ONE_LEVEL search to traverse the directory hierarchy, that is the
preferred way.
>> However if possible I'd avoid such a costly tree traversal, and instead
>> use e.g. paged search.
> Actually, it's probably better to abandon the request when you get what
> you want, paged search will just mitigate the memory used on the client
> side.
>
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 03/02/15 22:07, Stefan Seelmann a écrit :
> I forgot to mention the performance aspect.
>
> If you traverse all persons from the CEO down you need as many LDAP
> search operations as you have persons in the directory, each require a
> full network roundtrip, which takes time.
What's the point of doing that when a ONE_LEVEL search done one level
below would provide all the entries with one single Search ?
> However if possible I'd avoid such a costly tree traversal, and instead
> use e.g. paged search.
Actually, it's probably better to abandon the request when you get what
you want, paged search will just mitigate the memory used on the client
side.
Re: Proper use of LdapConnectionPool
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
I forgot to mention the performance aspect.
If you traverse all persons from the CEO down you need as many LDAP
search operations as you have persons in the directory, each require a
full network roundtrip, which takes time. If you want to parallelize
that, I'd recomment to convert the recursive tree traversal into some
iterative processing. Create a fixed thread pool of workers e.g. 4 or 8,
to limit the number of connections. Separate the production and
consumption of DNs by creating a thread-safe list. A worker takes one DN
from the list, retrieves the entry, and adds all "directreports" DNs
into the list, which can be taken by free workers.
However if possible I'd avoid such a costly tree traversal, and instead
use e.g. paged search.
Kind Regards,
Stefan
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 03/02/15 09:19, Harris, Christopher P a écrit :
> Stefan,
>
> Yes, we have a hierarchical structure.
>
> That's are fantastic ideas, Stefan! I especially like the search base idea.
FTR :
OBJECT scope : you get the object itself
ONELEVEL scope : youget all the children, but nor their descendants nor
the entry itself
SUBLEVEL : you get the entry, its children and all their descendants
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Stefan,
Yes, we have a hierarchical structure.
That's are fantastic ideas, Stefan! I especially like the search base idea.
I'll try them out.
Thank you!
- Chris
-----Original Message-----
From: Stefan Seelmann [mailto:mail@stefan-seelmann.de]
Sent: Tuesday, February 03, 2015 1:42 AM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Hi Harris,
can you tell a bit about your directory structure? Do you have a flat directory i.e. all Persons you are query for are below one OU, our do you have a nested hierarchical directory structure?
Looking at the code I guess the latter is the case, and then it is clear that onelevel search isn't working.
* You get the entry of the CEO.
* You get the "directreports" attribute which seems to contain DNs
* For each of those DNs you construct a new search, with the same search base, the search filter includes the one DN.
==> In that case I think you can use the DN directly as search base and search scope "object", because you already know the DN. The LDAP API also contains a "lookup" method that you can use.
Kind Regards,
Stefan
> ...
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase)); ...
> sb.append("(&(objectClass=person)(distinguishedName=")
> .append(replaceFilterSpecialCharactersWithHexRepresentation(distinguishedName))
> .append("))");
> ...
> .setDirectReports(entry.get( "directreports" ))
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Hi Harris,
can you tell a bit about your directory structure? Do you have a flat
directory i.e. all Persons you are query for are below one OU, our do
you have a nested hierarchical directory structure?
Looking at the code I guess the latter is the case, and then it is clear
that onelevel search isn't working.
* You get the entry of the CEO.
* You get the "directreports" attribute which seems to contain DNs
* For each of those DNs you construct a new search, with the same search
base, the search filter includes the one DN.
==> In that case I think you can use the DN directly as search base and
search scope "object", because you already know the DN. The LDAP API
also contains a "lookup" method that you can use.
Kind Regards,
Stefan
> ...
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> ...
> sb.append("(&(objectClass=person)(distinguishedName=")
> .append(replaceFilterSpecialCharactersWithHexRepresentation(distinguishedName))
> .append("))");
> ...
> .setDirectReports(entry.get( "directreports" ))
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Well...I can GET data, but not as expected.
So, yes. I agree...something seems amiss with this AD server. I may have to start asking our appointed AD people (or ultimately IBM - ha) for answers.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Monday, February 02, 2015 9:39 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 03/02/15 02:11, Harris, Christopher P a écrit :
> Hi, Kiran.
>
> I followed your advice and used the code from that page to construct and Ldap query using JNDI.
>
> The query only returns results when I specify SUBTREE, not when I use the other 2 scopes. I also tried all 3 scopes with referrals turned on and off with the same results.
Ok, now that makes more sense, kind of.
>
> Is there some setting in AD that only allows SUBTREE to be used for queries?
Not that we know.
>
> Still, SUBTREE would imply that an actual mass of person entries, instead of 1 person entries, would be returned if I searched for the CEO for example.
>
> I may just have to search using SUBTREE, expect it to incorrectly act like OBJECT_SCOPE, and execute a filter/query that returns every active employee account.
>
> Advice, thoughts?
It makes no sense... If you can't get data out of the AD server, either with the Apache LDAP API, or with JNDI, that means something is severely broken on your AD server...
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 03/02/15 02:11, Harris, Christopher P a écrit :
> Hi, Kiran.
>
> I followed your advice and used the code from that page to construct and Ldap query using JNDI.
>
> The query only returns results when I specify SUBTREE, not when I use the other 2 scopes. I also tried all 3 scopes with referrals turned on and off with the same results.
Ok, now that makes more sense, kind of.
>
> Is there some setting in AD that only allows SUBTREE to be used for queries?
Not that we know.
>
> Still, SUBTREE would imply that an actual mass of person entries, instead of 1 person entries, would be returned if I searched for the CEO for example.
>
> I may just have to search using SUBTREE, expect it to incorrectly act like OBJECT_SCOPE, and execute a filter/query that returns every active employee account.
>
> Advice, thoughts?
It makes no sense... If you can't get data out of the AD server, either
with the Apache LDAP API, or with JNDI, that means something is severely
broken on your AD server...
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Hi, Kiran.
I followed your advice and used the code from that page to construct and Ldap query using JNDI.
The query only returns results when I specify SUBTREE, not when I use the other 2 scopes. I also tried all 3 scopes with referrals turned on and off with the same results.
Is there some setting in AD that only allows SUBTREE to be used for queries?
Still, SUBTREE would imply that an actual mass of person entries, instead of 1 person entries, would be returned if I searched for the CEO for example.
I may just have to search using SUBTREE, expect it to incorrectly act like OBJECT_SCOPE, and execute a filter/query that returns every active employee account.
Advice, thoughts?
- Chris
-----Original Message-----
From: Kiran Ayyagari [mailto:kayyagari@apache.org]
Sent: Monday, February 02, 2015 5:49 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
On Tue, Feb 3, 2015 at 7:40 AM, Harris, Christopher P < chris_harris@baxter.com> wrote:
> Right. I understand the difference between the scopes.
>
> The problem is that the only scope that works for me against AD, in
> Apache Directory Studio and the LDAP API, is SUBTREE. The other 2
> scopes do not return results.
>
can you try with JNDI and see if you are able to fetch the RootDSE. See this[1] thread for a possible caveat while using JNDI.
Let us know if this works, then we need to check why it is not returned for the client using API
[1] https://community.oracle.com/thread/1157742
>
> - Chris
>
>
> -----Original Message-----
> From: Kiran Ayyagari [mailto:kayyagari@apache.org]
> Sent: Monday, February 02, 2015 5:30 PM
> To: api@directory.apache.org
> Subject: Re: Proper use of LdapConnectionPool
>
> On Wed, Jan 28, 2015 at 8:27 AM, Harris, Christopher P <
> chris_harris@baxter.com> wrote:
>
> > O.K. I think I remember why I stopped looking at the SearchScope.
> > If I just change the scope to either ONELEVEL or OBJECT, the cursor
> > comes back empty.
> >
> > Is this an AD "feature" or is this a bug in the API?
> >
> > OBJECT scope on rootDSE should work, and looks like AD allows it as
> > well
>
> > It looks like I'll need to do 1 query using the SUBTREE scope.
> >
> no, this will not solve the problem, you _must_ do ONE_LEVEL searches
> otherwise you either reach time or size limits enforced by AD.
>
> get rootDSE first, read the 'namingContexts' and then perform your
> search on each context separately, recursively using ONE level scope.
>
>
> > - Chris
> >
> > -----Original Message-----
> > From: Harris, Christopher P
> > Sent: Tuesday, January 27, 2015 6:16 PM
> > To: api@directory.apache.org
> > Subject: RE: Proper use of LdapConnectionPool
> >
> > Ah, crap. I forgot to look at the Scope. I've been using this code
> > for so long for single-search queries that I took it for granted.
> >
> > I'll try setting it to ONE_LEVEL to simply bask in the glory of the
> > speedy results, but still, doing just 1 query makes a hell of a lot
> > more
> sense.
> >
> > Sorry, I don't know where my head was.
> >
> > Thanks for steering me down the right path.
> >
> > - Chris
> >
> >
> > -----Original Message-----
> > From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> > Sent: Tuesday, January 27, 2015 5:43 PM
> > To: api@directory.apache.org
> > Subject: Re: Proper use of LdapConnectionPool
> >
> > Le 27/01/15 23:07, Harris, Christopher P a écrit :
> > > Hi, Emmanuel.
> > >
> > > "Can you tell us how you do that ? Ie, are you using a plain new
> > connection for each thread you spawn ?"
> > > Sure. I can tell you how I am implementing a multi-threaded
> > > approach to
> > read all of LDAP/AD into memory. I'll do the next best
> > thing...paste my code at the end of my response.
> > >
> > >
> > > "In any case, the TimeOut is the default LDapConnection timeout
> > > (30
> > seconds) :"
> > > Yes, I noticed mention of the default timeout in your User Guide.
> > >
> > >
> > > "You have to set the LdapConnectionConfig timeout for all the
> > > created
> > connections to use it. there is a setTimeout() method for that which
> > has been added in 1.0.0-M28."
> > > When visiting your site while seeking to explore connection pool
> > options, I noticed that you recently released M28 and fixed
> > DIRAPI-217 and decided to update my pom.xml to M28 and test out the
> > PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
> > >
> > > Oh, and your example needs to be updated to using
> > DefaultPoolableLdapConnectionFactory instead of
> > PoolableLdapConnectionFactory.
> > >
> > >
> > > "config.setTimeOut( whatever fits you );"
> > > Very good to know. Thank you!
> > >
> > >
> > > "It is the right way."
> > > Sweeeeeeet!
> > >
> > >
> > > "Side note : you may face various problems when pulling everything
> > > from an AD server. Typically, the AD config might not let you pull
> > > more than
> > > 1000 entries, as there is a hard limit you need to change on AD if
> > > you
> > want to get more entries.
> > >
> > > Otherwise, the approach - ie, using multiple threads - might seems
> > > good,
> > but the benefit is limited. Pulling entries from the server is fast,
> > you should be able to get tens of thousands per second with one
> > single
> thread.
> > I'm not sure how AD support concurrent searches anyway. Last, not
> > least, it's likely that AD does not allow more than a certain number
> > of concurrent threads to run, which might lead to contention at some
> point."
> > >
> > > Ah, this is why I wanted to reach out to you guys. You guys know
> > > this
> > kind of in-depth information about LDAP and AD. So, I may adapt my
> > code to a single-thread then. I can live with that. I need to pull
> > about 40k-60k entries, so 10's of thousands of entries per second
> > works for me. I may need to run the code by you then if I go with a
> > single-threaded approach and need to check if I'm going about it in
> > the
> most efficient manner.
> >
> > The pb with the multi-threaded approach is that you *have* to know
> > which entry has children, because they won't give you such an info.
> > So you will end doing a search for every single entry you get at one
> > level, with scope ONE_LEVEL, and most of the time, you will just get
> > teh entry itself. That would more than double the time it takes to
> > grab
> everything.
> >
> > >
> > >
> > >
> > > And now time for some code...
> > >
> > > import java.io.IOException;
> > > import java.util.Iterator;
> > > import java.util.List;
> > > import java.util.Map;
> > > import java.util.concurrent.ConcurrentHashMap;
> > > import java.util.concurrent.ExecutorService;
> > > import java.util.concurrent.Executors; import
> > > java.util.concurrent.TimeUnit; import java.util.logging.Level;
> > > import java.util.logging.Logger;
> > >
> > > import org.apache.commons.pool.impl.GenericObjectPool;
> > > import org.apache.directory.api.ldap.model.cursor.CursorException;
> > > import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> > > import org.apache.directory.api.ldap.model.entry.Entry;
> > > import
> > > org.apache.directory.api.ldap.model.exception.LdapException;
> > > import org.apache.directory.api.ldap.model.message.Response;
> > > import org.apache.directory.api.ldap.model.message.SearchRequest;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchResultEntry;
> > > import org.apache.directory.api.ldap.model.message.SearchScope;
> > > import org.apache.directory.api.ldap.model.name.Dn;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> > > import org.apache.directory.ldap.client.api.LdapConnection;
> > > import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> > > import org.apache.directory.ldap.client.api.LdapConnectionPool;
> > > import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultPoolableLdapConnection
> > > Fa
> > > ct
> > > ory; import
> > > org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnect
> > > io
> > > nF actory; import
> > > org.apache.directory.ldap.client.api.SearchCursorImpl;
> > > import org.apache.directory.ldap.client.template.EntryMapper;
> > > import
> > > org.apache.directory.ldap.client.template.LdapConnectionTemplate;
> > >
> > > /**
> > > * @author Chris Harris
> > > *
> > > */
> > > public class LdapClient {
> > >
> > > public LdapClient() {
> > >
> > > }
> > >
> > > public Person searchLdapForCeo() {
> > > return this.searchLdapUsingHybridApproach(ceoQuery);
> > > }
> > >
> > > public Map<String, Person> buildLdapMap() {
> > > SearchCursor cursor = new SearchCursorImpl(null,
> > > 300000,
> > TimeUnit.SECONDS);
> > > LdapConnection connection = new
> > LdapNetworkConnection(host, port);
> > > connection.setTimeOut(300000);
> > > Entry entry = null;
> > >
> > > try {
> > > connection.bind(dn, pwd);
> > >
> > LdapClient.recursivelyGetLdapDirectReports(connection, cursor,
> > entry, ceoQuery);
> > > System.out.println("Finished all
> > > Ldap Map
> > Builder threads...");
> > > } catch (LdapException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE,
> > null, ex);
> > > } catch (CursorException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE,
> > null, ex);
> > > } finally {
> > > cursor.close();
> > > try {
> > > connection.close();
> > > } catch (IOException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE,
> > null, ex);
> > > }
> > > }
> > >
> > > return concurrentPersonMap;
> > > }
> > >
> > > private static Person
> > recursivelyGetLdapDirectReports(LdapConnection connection,
> > SearchCursor cursor, Entry entry, String query)
> > > throws CursorException {
> > > Person p = null;
> > > EntryMapper<Person> em =
> > > Person.getEntryMapper();
> > >
> > > try {
> > > SearchRequest sr = new
> SearchRequestImpl();
> > > sr.setBase(new Dn(searchBase));
> > > StringBuilder sb = new
> > StringBuilder(query);
> > > sr.setFilter(sb.toString());
> > > sr.setScope( SearchScope.SUBTREE );
> >
> > Ahhhhh !!!! STOP !!!
> >
> > Ok, no need to go any further in your code.
> >
> > You are doing a SUBTREE search on *every single entry* you are
> > pulling from the base. if you have 40 000 entries, you will do
> > something like O(
> > 40 000! ) (factorial) searches. No wonder why you get timeout...
> > Imagine you have such a tree :
> >
> > root
> > A1
> > B1
> > C1
> > C2
> > B2
> > C3
> > C4
> > A2
> > B3
> > C5
> > C6
> > B4
> > C7
> > C8
> >
> > The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14
> > entries
> > -> 14 searches)
> > Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries ->
> > 6
> > searches)
> > Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries ->
> > 6
> > searches)
> > Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4
> > =
> > 8 ...
> >
> > At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you
> > have only 15 entries...
> >
> > If you want to see what your algorithm is doing, just do a search
> > using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40
> > 000) searches, which is way less than what you are doing.
> >
> > But anyway, doing a search on the root with a SUBTREE scope will be
> > way faster, because you will do only one single search.
> >
> >
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the
> > sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the
> > sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
> The information transmitted is intended only for the person(s) or
> entity to which it is addressed and may contain confidential and/or
> legally privileged material. Delivery of this message to any person
> other than the intended recipient(s) is not intended in any way to
> waive privilege or confidentiality. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance
> upon, this information by entities other than the intended recipient
> is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
>
--
Kiran Ayyagari
http://keydap.com
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Kiran Ayyagari <ka...@apache.org>.
On Tue, Feb 3, 2015 at 7:40 AM, Harris, Christopher P <
chris_harris@baxter.com> wrote:
> Right. I understand the difference between the scopes.
>
> The problem is that the only scope that works for me against AD, in Apache
> Directory Studio and the LDAP API, is SUBTREE. The other 2 scopes do not
> return results.
>
can you try with JNDI and see if you are able to fetch the RootDSE. See
this[1] thread
for a possible caveat while using JNDI.
Let us know if this works, then we need to check why it is not returned for
the client using API
[1] https://community.oracle.com/thread/1157742
>
> - Chris
>
>
> -----Original Message-----
> From: Kiran Ayyagari [mailto:kayyagari@apache.org]
> Sent: Monday, February 02, 2015 5:30 PM
> To: api@directory.apache.org
> Subject: Re: Proper use of LdapConnectionPool
>
> On Wed, Jan 28, 2015 at 8:27 AM, Harris, Christopher P <
> chris_harris@baxter.com> wrote:
>
> > O.K. I think I remember why I stopped looking at the SearchScope. If
> > I just change the scope to either ONELEVEL or OBJECT, the cursor comes
> > back empty.
> >
> > Is this an AD "feature" or is this a bug in the API?
> >
> > OBJECT scope on rootDSE should work, and looks like AD allows it as
> > well
>
> > It looks like I'll need to do 1 query using the SUBTREE scope.
> >
> no, this will not solve the problem, you _must_ do ONE_LEVEL searches
> otherwise you either reach time or size limits enforced by AD.
>
> get rootDSE first, read the 'namingContexts' and then perform your search
> on each context separately, recursively using ONE level scope.
>
>
> > - Chris
> >
> > -----Original Message-----
> > From: Harris, Christopher P
> > Sent: Tuesday, January 27, 2015 6:16 PM
> > To: api@directory.apache.org
> > Subject: RE: Proper use of LdapConnectionPool
> >
> > Ah, crap. I forgot to look at the Scope. I've been using this code
> > for so long for single-search queries that I took it for granted.
> >
> > I'll try setting it to ONE_LEVEL to simply bask in the glory of the
> > speedy results, but still, doing just 1 query makes a hell of a lot more
> sense.
> >
> > Sorry, I don't know where my head was.
> >
> > Thanks for steering me down the right path.
> >
> > - Chris
> >
> >
> > -----Original Message-----
> > From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> > Sent: Tuesday, January 27, 2015 5:43 PM
> > To: api@directory.apache.org
> > Subject: Re: Proper use of LdapConnectionPool
> >
> > Le 27/01/15 23:07, Harris, Christopher P a écrit :
> > > Hi, Emmanuel.
> > >
> > > "Can you tell us how you do that ? Ie, are you using a plain new
> > connection for each thread you spawn ?"
> > > Sure. I can tell you how I am implementing a multi-threaded
> > > approach to
> > read all of LDAP/AD into memory. I'll do the next best thing...paste
> > my code at the end of my response.
> > >
> > >
> > > "In any case, the TimeOut is the default LDapConnection timeout (30
> > seconds) :"
> > > Yes, I noticed mention of the default timeout in your User Guide.
> > >
> > >
> > > "You have to set the LdapConnectionConfig timeout for all the
> > > created
> > connections to use it. there is a setTimeout() method for that which
> > has been added in 1.0.0-M28."
> > > When visiting your site while seeking to explore connection pool
> > options, I noticed that you recently released M28 and fixed DIRAPI-217
> > and decided to update my pom.xml to M28 and test out the
> > PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
> > >
> > > Oh, and your example needs to be updated to using
> > DefaultPoolableLdapConnectionFactory instead of
> > PoolableLdapConnectionFactory.
> > >
> > >
> > > "config.setTimeOut( whatever fits you );"
> > > Very good to know. Thank you!
> > >
> > >
> > > "It is the right way."
> > > Sweeeeeeet!
> > >
> > >
> > > "Side note : you may face various problems when pulling everything
> > > from an AD server. Typically, the AD config might not let you pull
> > > more than
> > > 1000 entries, as there is a hard limit you need to change on AD if
> > > you
> > want to get more entries.
> > >
> > > Otherwise, the approach - ie, using multiple threads - might seems
> > > good,
> > but the benefit is limited. Pulling entries from the server is fast,
> > you should be able to get tens of thousands per second with one single
> thread.
> > I'm not sure how AD support concurrent searches anyway. Last, not
> > least, it's likely that AD does not allow more than a certain number
> > of concurrent threads to run, which might lead to contention at some
> point."
> > >
> > > Ah, this is why I wanted to reach out to you guys. You guys know
> > > this
> > kind of in-depth information about LDAP and AD. So, I may adapt my
> > code to a single-thread then. I can live with that. I need to pull
> > about 40k-60k entries, so 10's of thousands of entries per second
> > works for me. I may need to run the code by you then if I go with a
> > single-threaded approach and need to check if I'm going about it in the
> most efficient manner.
> >
> > The pb with the multi-threaded approach is that you *have* to know
> > which entry has children, because they won't give you such an info. So
> > you will end doing a search for every single entry you get at one
> > level, with scope ONE_LEVEL, and most of the time, you will just get
> > teh entry itself. That would more than double the time it takes to grab
> everything.
> >
> > >
> > >
> > >
> > > And now time for some code...
> > >
> > > import java.io.IOException;
> > > import java.util.Iterator;
> > > import java.util.List;
> > > import java.util.Map;
> > > import java.util.concurrent.ConcurrentHashMap;
> > > import java.util.concurrent.ExecutorService;
> > > import java.util.concurrent.Executors; import
> > > java.util.concurrent.TimeUnit; import java.util.logging.Level;
> > > import java.util.logging.Logger;
> > >
> > > import org.apache.commons.pool.impl.GenericObjectPool;
> > > import org.apache.directory.api.ldap.model.cursor.CursorException;
> > > import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> > > import org.apache.directory.api.ldap.model.entry.Entry;
> > > import org.apache.directory.api.ldap.model.exception.LdapException;
> > > import org.apache.directory.api.ldap.model.message.Response;
> > > import org.apache.directory.api.ldap.model.message.SearchRequest;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchResultEntry;
> > > import org.apache.directory.api.ldap.model.message.SearchScope;
> > > import org.apache.directory.api.ldap.model.name.Dn;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> > > import org.apache.directory.ldap.client.api.LdapConnection;
> > > import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> > > import org.apache.directory.ldap.client.api.LdapConnectionPool;
> > > import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFa
> > > ct
> > > ory; import
> > > org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectio
> > > nF actory; import
> > > org.apache.directory.ldap.client.api.SearchCursorImpl;
> > > import org.apache.directory.ldap.client.template.EntryMapper;
> > > import
> > > org.apache.directory.ldap.client.template.LdapConnectionTemplate;
> > >
> > > /**
> > > * @author Chris Harris
> > > *
> > > */
> > > public class LdapClient {
> > >
> > > public LdapClient() {
> > >
> > > }
> > >
> > > public Person searchLdapForCeo() {
> > > return this.searchLdapUsingHybridApproach(ceoQuery);
> > > }
> > >
> > > public Map<String, Person> buildLdapMap() {
> > > SearchCursor cursor = new SearchCursorImpl(null,
> > > 300000,
> > TimeUnit.SECONDS);
> > > LdapConnection connection = new
> > LdapNetworkConnection(host, port);
> > > connection.setTimeOut(300000);
> > > Entry entry = null;
> > >
> > > try {
> > > connection.bind(dn, pwd);
> > >
> > LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry,
> > ceoQuery);
> > > System.out.println("Finished all Ldap
> > > Map
> > Builder threads...");
> > > } catch (LdapException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > > } catch (CursorException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > > } finally {
> > > cursor.close();
> > > try {
> > > connection.close();
> > > } catch (IOException ex) {
> > >
> > Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > > }
> > > }
> > >
> > > return concurrentPersonMap;
> > > }
> > >
> > > private static Person
> > recursivelyGetLdapDirectReports(LdapConnection connection,
> > SearchCursor cursor, Entry entry, String query)
> > > throws CursorException {
> > > Person p = null;
> > > EntryMapper<Person> em =
> > > Person.getEntryMapper();
> > >
> > > try {
> > > SearchRequest sr = new
> SearchRequestImpl();
> > > sr.setBase(new Dn(searchBase));
> > > StringBuilder sb = new
> > StringBuilder(query);
> > > sr.setFilter(sb.toString());
> > > sr.setScope( SearchScope.SUBTREE );
> >
> > Ahhhhh !!!! STOP !!!
> >
> > Ok, no need to go any further in your code.
> >
> > You are doing a SUBTREE search on *every single entry* you are pulling
> > from the base. if you have 40 000 entries, you will do something like
> > O(
> > 40 000! ) (factorial) searches. No wonder why you get timeout...
> > Imagine you have such a tree :
> >
> > root
> > A1
> > B1
> > C1
> > C2
> > B2
> > C3
> > C4
> > A2
> > B3
> > C5
> > C6
> > B4
> > C7
> > C8
> >
> > The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
> > -> 14 searches)
> > Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
> > searches)
> > Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
> > searches)
> > Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 =
> > 8 ...
> >
> > At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have
> > only 15 entries...
> >
> > If you want to see what your algorithm is doing, just do a search
> > using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40
> > 000) searches, which is way less than what you are doing.
> >
> > But anyway, doing a search on the root with a SUBTREE scope will be
> > way faster, because you will do only one single search.
> >
> >
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
> The information transmitted is intended only for the person(s) or entity
> to which it is addressed and may contain confidential and/or legally
> privileged material. Delivery of this message to any person other than the
> intended recipient(s) is not intended in any way to waive privilege or
> confidentiality. Any review, retransmission, dissemination or other use of,
> or taking of any action in reliance upon, this information by entities
> other than the intended recipient is prohibited. If you receive this in
> error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
>
--
Kiran Ayyagari
http://keydap.com
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Right. I understand the difference between the scopes.
The problem is that the only scope that works for me against AD, in Apache Directory Studio and the LDAP API, is SUBTREE. The other 2 scopes do not return results.
- Chris
-----Original Message-----
From: Kiran Ayyagari [mailto:kayyagari@apache.org]
Sent: Monday, February 02, 2015 5:30 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
On Wed, Jan 28, 2015 at 8:27 AM, Harris, Christopher P < chris_harris@baxter.com> wrote:
> O.K. I think I remember why I stopped looking at the SearchScope. If
> I just change the scope to either ONELEVEL or OBJECT, the cursor comes
> back empty.
>
> Is this an AD "feature" or is this a bug in the API?
>
> OBJECT scope on rootDSE should work, and looks like AD allows it as
> well
> It looks like I'll need to do 1 query using the SUBTREE scope.
>
no, this will not solve the problem, you _must_ do ONE_LEVEL searches otherwise you either reach time or size limits enforced by AD.
get rootDSE first, read the 'namingContexts' and then perform your search on each context separately, recursively using ONE level scope.
> - Chris
>
> -----Original Message-----
> From: Harris, Christopher P
> Sent: Tuesday, January 27, 2015 6:16 PM
> To: api@directory.apache.org
> Subject: RE: Proper use of LdapConnectionPool
>
> Ah, crap. I forgot to look at the Scope. I've been using this code
> for so long for single-search queries that I took it for granted.
>
> I'll try setting it to ONE_LEVEL to simply bask in the glory of the
> speedy results, but still, doing just 1 query makes a hell of a lot more sense.
>
> Sorry, I don't know where my head was.
>
> Thanks for steering me down the right path.
>
> - Chris
>
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Tuesday, January 27, 2015 5:43 PM
> To: api@directory.apache.org
> Subject: Re: Proper use of LdapConnectionPool
>
> Le 27/01/15 23:07, Harris, Christopher P a écrit :
> > Hi, Emmanuel.
> >
> > "Can you tell us how you do that ? Ie, are you using a plain new
> connection for each thread you spawn ?"
> > Sure. I can tell you how I am implementing a multi-threaded
> > approach to
> read all of LDAP/AD into memory. I'll do the next best thing...paste
> my code at the end of my response.
> >
> >
> > "In any case, the TimeOut is the default LDapConnection timeout (30
> seconds) :"
> > Yes, I noticed mention of the default timeout in your User Guide.
> >
> >
> > "You have to set the LdapConnectionConfig timeout for all the
> > created
> connections to use it. there is a setTimeout() method for that which
> has been added in 1.0.0-M28."
> > When visiting your site while seeking to explore connection pool
> options, I noticed that you recently released M28 and fixed DIRAPI-217
> and decided to update my pom.xml to M28 and test out the
> PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
> >
> > Oh, and your example needs to be updated to using
> DefaultPoolableLdapConnectionFactory instead of
> PoolableLdapConnectionFactory.
> >
> >
> > "config.setTimeOut( whatever fits you );"
> > Very good to know. Thank you!
> >
> >
> > "It is the right way."
> > Sweeeeeeet!
> >
> >
> > "Side note : you may face various problems when pulling everything
> > from an AD server. Typically, the AD config might not let you pull
> > more than
> > 1000 entries, as there is a hard limit you need to change on AD if
> > you
> want to get more entries.
> >
> > Otherwise, the approach - ie, using multiple threads - might seems
> > good,
> but the benefit is limited. Pulling entries from the server is fast,
> you should be able to get tens of thousands per second with one single thread.
> I'm not sure how AD support concurrent searches anyway. Last, not
> least, it's likely that AD does not allow more than a certain number
> of concurrent threads to run, which might lead to contention at some point."
> >
> > Ah, this is why I wanted to reach out to you guys. You guys know
> > this
> kind of in-depth information about LDAP and AD. So, I may adapt my
> code to a single-thread then. I can live with that. I need to pull
> about 40k-60k entries, so 10's of thousands of entries per second
> works for me. I may need to run the code by you then if I go with a
> single-threaded approach and need to check if I'm going about it in the most efficient manner.
>
> The pb with the multi-threaded approach is that you *have* to know
> which entry has children, because they won't give you such an info. So
> you will end doing a search for every single entry you get at one
> level, with scope ONE_LEVEL, and most of the time, you will just get
> teh entry itself. That would more than double the time it takes to grab everything.
>
> >
> >
> >
> > And now time for some code...
> >
> > import java.io.IOException;
> > import java.util.Iterator;
> > import java.util.List;
> > import java.util.Map;
> > import java.util.concurrent.ConcurrentHashMap;
> > import java.util.concurrent.ExecutorService;
> > import java.util.concurrent.Executors; import
> > java.util.concurrent.TimeUnit; import java.util.logging.Level;
> > import java.util.logging.Logger;
> >
> > import org.apache.commons.pool.impl.GenericObjectPool;
> > import org.apache.directory.api.ldap.model.cursor.CursorException;
> > import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> > import org.apache.directory.api.ldap.model.entry.Entry;
> > import org.apache.directory.api.ldap.model.exception.LdapException;
> > import org.apache.directory.api.ldap.model.message.Response;
> > import org.apache.directory.api.ldap.model.message.SearchRequest;
> > import
> > org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> > import
> > org.apache.directory.api.ldap.model.message.SearchResultEntry;
> > import org.apache.directory.api.ldap.model.message.SearchScope;
> > import org.apache.directory.api.ldap.model.name.Dn;
> > import
> > org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> > import org.apache.directory.ldap.client.api.LdapConnection;
> > import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> > import org.apache.directory.ldap.client.api.LdapConnectionPool;
> > import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> > import
> > org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFa
> > ct
> > ory; import
> > org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectio
> > nF actory; import
> > org.apache.directory.ldap.client.api.SearchCursorImpl;
> > import org.apache.directory.ldap.client.template.EntryMapper;
> > import
> > org.apache.directory.ldap.client.template.LdapConnectionTemplate;
> >
> > /**
> > * @author Chris Harris
> > *
> > */
> > public class LdapClient {
> >
> > public LdapClient() {
> >
> > }
> >
> > public Person searchLdapForCeo() {
> > return this.searchLdapUsingHybridApproach(ceoQuery);
> > }
> >
> > public Map<String, Person> buildLdapMap() {
> > SearchCursor cursor = new SearchCursorImpl(null,
> > 300000,
> TimeUnit.SECONDS);
> > LdapConnection connection = new
> LdapNetworkConnection(host, port);
> > connection.setTimeOut(300000);
> > Entry entry = null;
> >
> > try {
> > connection.bind(dn, pwd);
> >
> LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry,
> ceoQuery);
> > System.out.println("Finished all Ldap
> > Map
> Builder threads...");
> > } catch (LdapException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> ex);
> > } catch (CursorException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> ex);
> > } finally {
> > cursor.close();
> > try {
> > connection.close();
> > } catch (IOException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> ex);
> > }
> > }
> >
> > return concurrentPersonMap;
> > }
> >
> > private static Person
> recursivelyGetLdapDirectReports(LdapConnection connection,
> SearchCursor cursor, Entry entry, String query)
> > throws CursorException {
> > Person p = null;
> > EntryMapper<Person> em =
> > Person.getEntryMapper();
> >
> > try {
> > SearchRequest sr = new SearchRequestImpl();
> > sr.setBase(new Dn(searchBase));
> > StringBuilder sb = new
> StringBuilder(query);
> > sr.setFilter(sb.toString());
> > sr.setScope( SearchScope.SUBTREE );
>
> Ahhhhh !!!! STOP !!!
>
> Ok, no need to go any further in your code.
>
> You are doing a SUBTREE search on *every single entry* you are pulling
> from the base. if you have 40 000 entries, you will do something like
> O(
> 40 000! ) (factorial) searches. No wonder why you get timeout...
> Imagine you have such a tree :
>
> root
> A1
> B1
> C1
> C2
> B2
> C3
> C4
> A2
> B3
> C5
> C6
> B4
> C7
> C8
>
> The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
> -> 14 searches)
> Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
> searches)
> Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
> searches)
> Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 =
> 8 ...
>
> At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have
> only 15 entries...
>
> If you want to see what your algorithm is doing, just do a search
> using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40
> 000) searches, which is way less than what you are doing.
>
> But anyway, doing a search on the root with a SUBTREE scope will be
> way faster, because you will do only one single search.
>
>
> The information transmitted is intended only for the person(s) or
> entity to which it is addressed and may contain confidential and/or
> legally privileged material. Delivery of this message to any person
> other than the intended recipient(s) is not intended in any way to
> waive privilege or confidentiality. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance
> upon, this information by entities other than the intended recipient
> is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
> The information transmitted is intended only for the person(s) or
> entity to which it is addressed and may contain confidential and/or
> legally privileged material. Delivery of this message to any person
> other than the intended recipient(s) is not intended in any way to
> waive privilege or confidentiality. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance
> upon, this information by entities other than the intended recipient
> is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
>
--
Kiran Ayyagari
http://keydap.com
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Jan 28, 2015 at 8:27 AM, Harris, Christopher P <
chris_harris@baxter.com> wrote:
> O.K. I think I remember why I stopped looking at the SearchScope. If I
> just change the scope to either ONELEVEL or OBJECT, the cursor comes back
> empty.
>
> Is this an AD "feature" or is this a bug in the API?
>
> OBJECT scope on rootDSE should work, and looks like AD allows it as well
> It looks like I'll need to do 1 query using the SUBTREE scope.
>
no, this will not solve the problem, you _must_ do ONE_LEVEL searches
otherwise you either reach time or size limits enforced by AD.
get rootDSE first, read the 'namingContexts' and then perform your search on
each context separately, recursively using ONE level scope.
> - Chris
>
> -----Original Message-----
> From: Harris, Christopher P
> Sent: Tuesday, January 27, 2015 6:16 PM
> To: api@directory.apache.org
> Subject: RE: Proper use of LdapConnectionPool
>
> Ah, crap. I forgot to look at the Scope. I've been using this code for
> so long for single-search queries that I took it for granted.
>
> I'll try setting it to ONE_LEVEL to simply bask in the glory of the speedy
> results, but still, doing just 1 query makes a hell of a lot more sense.
>
> Sorry, I don't know where my head was.
>
> Thanks for steering me down the right path.
>
> - Chris
>
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Tuesday, January 27, 2015 5:43 PM
> To: api@directory.apache.org
> Subject: Re: Proper use of LdapConnectionPool
>
> Le 27/01/15 23:07, Harris, Christopher P a écrit :
> > Hi, Emmanuel.
> >
> > "Can you tell us how you do that ? Ie, are you using a plain new
> connection for each thread you spawn ?"
> > Sure. I can tell you how I am implementing a multi-threaded approach to
> read all of LDAP/AD into memory. I'll do the next best thing...paste my
> code at the end of my response.
> >
> >
> > "In any case, the TimeOut is the default LDapConnection timeout (30
> seconds) :"
> > Yes, I noticed mention of the default timeout in your User Guide.
> >
> >
> > "You have to set the LdapConnectionConfig timeout for all the created
> connections to use it. there is a setTimeout() method for that which has
> been added in 1.0.0-M28."
> > When visiting your site while seeking to explore connection pool
> options, I noticed that you recently released M28 and fixed DIRAPI-217 and
> decided to update my pom.xml to M28 and test out the
> PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
> >
> > Oh, and your example needs to be updated to using
> DefaultPoolableLdapConnectionFactory instead of
> PoolableLdapConnectionFactory.
> >
> >
> > "config.setTimeOut( whatever fits you );"
> > Very good to know. Thank you!
> >
> >
> > "It is the right way."
> > Sweeeeeeet!
> >
> >
> > "Side note : you may face various problems when pulling everything
> > from an AD server. Typically, the AD config might not let you pull
> > more than
> > 1000 entries, as there is a hard limit you need to change on AD if you
> want to get more entries.
> >
> > Otherwise, the approach - ie, using multiple threads - might seems good,
> but the benefit is limited. Pulling entries from the server is fast, you
> should be able to get tens of thousands per second with one single thread.
> I'm not sure how AD support concurrent searches anyway. Last, not least,
> it's likely that AD does not allow more than a certain number of concurrent
> threads to run, which might lead to contention at some point."
> >
> > Ah, this is why I wanted to reach out to you guys. You guys know this
> kind of in-depth information about LDAP and AD. So, I may adapt my code to
> a single-thread then. I can live with that. I need to pull about 40k-60k
> entries, so 10's of thousands of entries per second works for me. I may
> need to run the code by you then if I go with a single-threaded approach
> and need to check if I'm going about it in the most efficient manner.
>
> The pb with the multi-threaded approach is that you *have* to know which
> entry has children, because they won't give you such an info. So you will
> end doing a search for every single entry you get at one level, with scope
> ONE_LEVEL, and most of the time, you will just get teh entry itself. That
> would more than double the time it takes to grab everything.
>
> >
> >
> >
> > And now time for some code...
> >
> > import java.io.IOException;
> > import java.util.Iterator;
> > import java.util.List;
> > import java.util.Map;
> > import java.util.concurrent.ConcurrentHashMap;
> > import java.util.concurrent.ExecutorService;
> > import java.util.concurrent.Executors; import
> > java.util.concurrent.TimeUnit; import java.util.logging.Level; import
> > java.util.logging.Logger;
> >
> > import org.apache.commons.pool.impl.GenericObjectPool;
> > import org.apache.directory.api.ldap.model.cursor.CursorException;
> > import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> > import org.apache.directory.api.ldap.model.entry.Entry;
> > import org.apache.directory.api.ldap.model.exception.LdapException;
> > import org.apache.directory.api.ldap.model.message.Response;
> > import org.apache.directory.api.ldap.model.message.SearchRequest;
> > import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> > import org.apache.directory.api.ldap.model.message.SearchResultEntry;
> > import org.apache.directory.api.ldap.model.message.SearchScope;
> > import org.apache.directory.api.ldap.model.name.Dn;
> > import
> > org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> > import org.apache.directory.ldap.client.api.LdapConnection;
> > import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> > import org.apache.directory.ldap.client.api.LdapConnectionPool;
> > import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> > import
> > org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFact
> > ory; import
> > org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionF
> > actory; import org.apache.directory.ldap.client.api.SearchCursorImpl;
> > import org.apache.directory.ldap.client.template.EntryMapper;
> > import
> > org.apache.directory.ldap.client.template.LdapConnectionTemplate;
> >
> > /**
> > * @author Chris Harris
> > *
> > */
> > public class LdapClient {
> >
> > public LdapClient() {
> >
> > }
> >
> > public Person searchLdapForCeo() {
> > return this.searchLdapUsingHybridApproach(ceoQuery);
> > }
> >
> > public Map<String, Person> buildLdapMap() {
> > SearchCursor cursor = new SearchCursorImpl(null, 300000,
> TimeUnit.SECONDS);
> > LdapConnection connection = new
> LdapNetworkConnection(host, port);
> > connection.setTimeOut(300000);
> > Entry entry = null;
> >
> > try {
> > connection.bind(dn, pwd);
> >
> LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry,
> ceoQuery);
> > System.out.println("Finished all Ldap Map
> Builder threads...");
> > } catch (LdapException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> > } catch (CursorException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> > } finally {
> > cursor.close();
> > try {
> > connection.close();
> > } catch (IOException ex) {
> >
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> > }
> > }
> >
> > return concurrentPersonMap;
> > }
> >
> > private static Person
> recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor
> cursor, Entry entry, String query)
> > throws CursorException {
> > Person p = null;
> > EntryMapper<Person> em = Person.getEntryMapper();
> >
> > try {
> > SearchRequest sr = new SearchRequestImpl();
> > sr.setBase(new Dn(searchBase));
> > StringBuilder sb = new
> StringBuilder(query);
> > sr.setFilter(sb.toString());
> > sr.setScope( SearchScope.SUBTREE );
>
> Ahhhhh !!!! STOP !!!
>
> Ok, no need to go any further in your code.
>
> You are doing a SUBTREE search on *every single entry* you are pulling
> from the base. if you have 40 000 entries, you will do something like O(
> 40 000! ) (factorial) searches. No wonder why you get timeout... Imagine
> you have such a tree :
>
> root
> A1
> B1
> C1
> C2
> B2
> C3
> C4
> A2
> B3
> C5
> C6
> B4
> C7
> C8
>
> The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
> -> 14 searches)
> Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
> searches)
> Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
> searches)
> Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 = 8
> ...
>
> At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have
> only 15 entries...
>
> If you want to see what your algorithm is doing, just do a search using a
> SearchScope.ONE_LEVEL instead. You will only do somehow O(40 000) searches,
> which is way less than what you are doing.
>
> But anyway, doing a search on the root with a SUBTREE scope will be way
> faster, because you will do only one single search.
>
>
> The information transmitted is intended only for the person(s) or entity
> to which it is addressed and may contain confidential and/or legally
> privileged material. Delivery of this message to any person other than the
> intended recipient(s) is not intended in any way to waive privilege or
> confidentiality. Any review, retransmission, dissemination or other use of,
> or taking of any action in reliance upon, this information by entities
> other than the intended recipient is prohibited. If you receive this in
> error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
> The information transmitted is intended only for the person(s) or entity
> to which it is addressed and may contain confidential and/or legally
> privileged material. Delivery of this message to any person other than the
> intended recipient(s) is not intended in any way to waive privilege or
> confidentiality. Any review, retransmission, dissemination or other use of,
> or taking of any action in reliance upon, this information by entities
> other than the intended recipient is prohibited. If you receive this in
> error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
>
--
Kiran Ayyagari
http://keydap.com
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Hi, Emmanuel.
Yes, the following code does not return an Entry. Nothing is printed to my console when I use OBJECT. As soon as I change it to SUBTREE, 1 Entry is printed to my console.
I'm observing comparable behavior in Apache Directory Studio.
public Person searchLdapUsingHybridApproach(String query) {
SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
LdapConnection connection = new LdapNetworkConnection(host, port);
Entry entry = null;
Person p = null;
try {
connection.bind(dn, pwd);
SearchRequest sr = new SearchRequestImpl();
sr.setBase(new Dn(searchBase));
StringBuilder sb = new StringBuilder(query);
sr.setFilter(sb.toString());
sr.setScope( SearchScope.OBJECT );
cursor = connection.search(sr);
Response response;
while (cursor.next() && cursor.isEntry()) {
response = cursor.get();
System.out.println(((SearchResultEntry)response).getEntry());
entry = cursor.getEntry();
EntryMapper<Person> em = Person.getEntryMapper();
p = em.map(entry);
}
} catch (LdapException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} catch (CursorException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} finally {
cursor.close();
try {
connection.close();
} catch (IOException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
}
return p;
}
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, January 27, 2015 6:49 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 28/01/15 01:27, Harris, Christopher P a écrit :
> O.K. I think I remember why I stopped looking at the SearchScope. If I just change the scope to either ONELEVEL or OBJECT, the cursor comes back empty.
ONE_LEVEL applied on a terminal entry will return nothing. With OBJECT, if the entry exists, it should work.
If you can test the simplest possible code that grab an entry with OBJECT scope, and if it does not work, then it may be a bug, but then, I'd like to know about.
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Emmanuel,
I agree with you.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Monday, February 02, 2015 9:35 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 02/02/15 23:56, Harris, Christopher P a écrit :
> givenName is present in AD. That maps to our employees' first names.
>
> I just checked the Person mapping again. All the attributes defined in my Person object are populated when I inspect the person instance in the debugger.
My test was done with a server that obviously does not contain everything you expected to get, thus the NPE. What I was just saying is that the basic search works (ie, whetever scope I'm using, I'm receiving all the entries), and if there is a pb, it's not the API which is the cause of it.
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 02/02/15 23:56, Harris, Christopher P a écrit :
> givenName is present in AD. That maps to our employees' first names.
>
> I just checked the Person mapping again. All the attributes defined in my Person object are populated when I inspect the person instance in the debugger.
My test was done with a server that obviously does not contain
everything you expected to get, thus the NPE. What I was just saying is
that the basic search works (ie, whetever scope I'm using, I'm receiving
all the entries), and if there is a pb, it's not the API which is the
cause of it.
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
givenName is present in AD. That maps to our employees' first names.
I just checked the Person mapping again. All the attributes defined in my Person object are populated when I inspect the person instance in the debugger.
- Chris
-----Original Message-----
From: Harris, Christopher P
Sent: Monday, February 02, 2015 4:33 PM
To: api@directory.apache.org
Subject: RE: Proper use of LdapConnectionPool
O.K. I can try that.
However, this issue is reproducible in Apache Directory Studio for me.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Monday, February 02, 2015 4:22 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 02/02/15 21:28, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> Do you have any updates on this issue?
Sorry, was busy the last few days...
Here are my findings :
Basically, it works. I have used a slightly different version of the code :
public void searchLdapUsingHybridApproach()
{
SearchCursor cursor = new SearchCursorImpl( null, 30000, TimeUnit.SECONDS );
Entry entry = null;
try
{
SearchRequest sr = new SearchRequestImpl();
sr.setBase( new Dn( "ou=system" ) );
sr.setFilter( "(ObjectClass=*)" );
sr.setScope( SearchScope.SUBTREE );
cursor = connection.search( sr );
Response response;
while ( cursor.next() && cursor.isEntry() )
{
response = cursor.get();
System.out.println( ( ( SearchResultEntry ) response
).getEntry() );
entry = cursor.getEntry();
}
}
catch ( LdapException ex )
{
ex.printStackTrace();
}
catch ( CursorException ex )
{
ex.printStackTrace();
}
finally
{
cursor.close();
try
{
connection.close();
}
catch ( IOException ex )
{
ex.printStackTrace();
}
}
}
just to check that ther API returns the correct entries in the various cases (OBJECT, ONELEVEL and SUBTREE). It does.
So now, here is my hypothesys : there is a bug in the EntryMapper.
When I run the code using the EntryMapper, I get a NPE because the EntryMapper tries to do that :
public Person map( Entry entry ) throws LdapException
{
try
{
return new Person.Builder()
.setDistinguishedName( entry.getDn().getName() )
.setFirstName( entry.get( "givenName"
).getString() ) <---- NPE here because I have no givenName in the
entry I'm reading
.setLastName( entry.get( "sn" ).getString() )
.setTitle( entry.get( "title" ).getString() )
.setDepartment( entry.get( "department"
).getString() )
.setDivision( entry.get( "extensionAttribute14" ).getString() )
.setCity( entry.get( "l" ).getString() )
.setCountry( entry.get( "co" ).getString() )
.setEmail( ( entry.get( "mail" ) != null ) ?
entry.get( "mail" ).getString() : "" )
.setDeskLocation( entry.get( "postOfficeBox"
).getString() )
.setOfficePhone(
( entry.get( "telephoneNumber" ) != null
) ? entry.get( "telephoneNumber" ).getString()
: "" )
.setDirectReports( entry.get( "directreports" ) )
.build();
So my guess is that in your case your entry is missing some of the expected attribute, and this is why you get no result.
Can you chack that by printing the entry just hafter having pulled it from the LDAP server ?
Thanks !
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
O.K. I can try that.
However, this issue is reproducible in Apache Directory Studio for me.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Monday, February 02, 2015 4:22 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 02/02/15 21:28, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> Do you have any updates on this issue?
Sorry, was busy the last few days...
Here are my findings :
Basically, it works. I have used a slightly different version of the code :
public void searchLdapUsingHybridApproach()
{
SearchCursor cursor = new SearchCursorImpl( null, 30000, TimeUnit.SECONDS );
Entry entry = null;
try
{
SearchRequest sr = new SearchRequestImpl();
sr.setBase( new Dn( "ou=system" ) );
sr.setFilter( "(ObjectClass=*)" );
sr.setScope( SearchScope.SUBTREE );
cursor = connection.search( sr );
Response response;
while ( cursor.next() && cursor.isEntry() )
{
response = cursor.get();
System.out.println( ( ( SearchResultEntry ) response
).getEntry() );
entry = cursor.getEntry();
}
}
catch ( LdapException ex )
{
ex.printStackTrace();
}
catch ( CursorException ex )
{
ex.printStackTrace();
}
finally
{
cursor.close();
try
{
connection.close();
}
catch ( IOException ex )
{
ex.printStackTrace();
}
}
}
just to check that ther API returns the correct entries in the various cases (OBJECT, ONELEVEL and SUBTREE). It does.
So now, here is my hypothesys : there is a bug in the EntryMapper.
When I run the code using the EntryMapper, I get a NPE because the EntryMapper tries to do that :
public Person map( Entry entry ) throws LdapException
{
try
{
return new Person.Builder()
.setDistinguishedName( entry.getDn().getName() )
.setFirstName( entry.get( "givenName"
).getString() ) <---- NPE here because I have no givenName in the
entry I'm reading
.setLastName( entry.get( "sn" ).getString() )
.setTitle( entry.get( "title" ).getString() )
.setDepartment( entry.get( "department"
).getString() )
.setDivision( entry.get( "extensionAttribute14" ).getString() )
.setCity( entry.get( "l" ).getString() )
.setCountry( entry.get( "co" ).getString() )
.setEmail( ( entry.get( "mail" ) != null ) ?
entry.get( "mail" ).getString() : "" )
.setDeskLocation( entry.get( "postOfficeBox"
).getString() )
.setOfficePhone(
( entry.get( "telephoneNumber" ) != null
) ? entry.get( "telephoneNumber" ).getString()
: "" )
.setDirectReports( entry.get( "directreports" ) )
.build();
So my guess is that in your case your entry is missing some of the expected attribute, and this is why you get no result.
Can you chack that by printing the entry just hafter having pulled it from the LDAP server ?
Thanks !
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 02/02/15 21:28, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> Do you have any updates on this issue?
Sorry, was busy the last few days...
Here are my findings :
Basically, it works. I have used a slightly different version of the code :
public void searchLdapUsingHybridApproach()
{
SearchCursor cursor = new SearchCursorImpl( null, 30000,
TimeUnit.SECONDS );
Entry entry = null;
try
{
SearchRequest sr = new SearchRequestImpl();
sr.setBase( new Dn( "ou=system" ) );
sr.setFilter( "(ObjectClass=*)" );
sr.setScope( SearchScope.SUBTREE );
cursor = connection.search( sr );
Response response;
while ( cursor.next() && cursor.isEntry() )
{
response = cursor.get();
System.out.println( ( ( SearchResultEntry ) response
).getEntry() );
entry = cursor.getEntry();
}
}
catch ( LdapException ex )
{
ex.printStackTrace();
}
catch ( CursorException ex )
{
ex.printStackTrace();
}
finally
{
cursor.close();
try
{
connection.close();
}
catch ( IOException ex )
{
ex.printStackTrace();
}
}
}
just to check that ther API returns the correct entries in the various
cases (OBJECT, ONELEVEL and SUBTREE). It does.
So now, here is my hypothesys : there is a bug in the EntryMapper.
When I run the code using the EntryMapper, I get a NPE because the
EntryMapper tries to do that :
public Person map( Entry entry ) throws LdapException
{
try
{
return new Person.Builder()
.setDistinguishedName( entry.getDn().getName() )
.setFirstName( entry.get( "givenName"
).getString() ) <---- NPE here because I have no givenName in the
entry I'm reading
.setLastName( entry.get( "sn" ).getString() )
.setTitle( entry.get( "title" ).getString() )
.setDepartment( entry.get( "department"
).getString() )
.setDivision( entry.get(
"extensionAttribute14" ).getString() )
.setCity( entry.get( "l" ).getString() )
.setCountry( entry.get( "co" ).getString() )
.setEmail( ( entry.get( "mail" ) != null ) ?
entry.get( "mail" ).getString() : "" )
.setDeskLocation( entry.get( "postOfficeBox"
).getString() )
.setOfficePhone(
( entry.get( "telephoneNumber" ) != null
) ? entry.get( "telephoneNumber" ).getString()
: "" )
.setDirectReports( entry.get(
"directreports" ) )
.build();
So my guess is that in your case your entry is missing some of the
expected attribute, and this is why you get no result.
Can you chack that by printing the entry just hafter having pulled it
from the LDAP server ?
Thanks !
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Hi, Emmanuel.
Do you have any updates on this issue?
I really want to use the LDAP API, but this is a showstopper for me that leaves me with 2 choices:
1.) Use another API. I'd rather not but will do it if I have no other option.
2.) If you have no one to test against AD, then I'll attempt to fix the problem myself and provide a patch. However, I need pointers on which .java files I should concentrate.
- Chris
-----Original Message-----
From: Harris, Christopher P
Sent: Wednesday, January 28, 2015 2:03 AM
To: 'api@directory.apache.org'
Subject: RE: Proper use of LdapConnectionPool
Hi, Emmanuel.
Yes, the following code does not return an Entry. Nothing is printed to my console when I use OBJECT. As soon as I change it to SUBTREE, 1 Entry is printed to my console.
I'm observing comparable behavior in Apache Directory Studio.
public Person searchLdapUsingHybridApproach(String query) {
SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
LdapConnection connection = new LdapNetworkConnection(host, port);
Entry entry = null;
Person p = null;
try {
connection.bind(dn, pwd);
SearchRequest sr = new SearchRequestImpl();
sr.setBase(new Dn(searchBase));
StringBuilder sb = new StringBuilder(query);
sr.setFilter(sb.toString());
sr.setScope( SearchScope.OBJECT );
cursor = connection.search(sr);
Response response;
while (cursor.next() && cursor.isEntry()) {
response = cursor.get();
System.out.println(((SearchResultEntry)response).getEntry());
entry = cursor.getEntry();
EntryMapper<Person> em = Person.getEntryMapper();
p = em.map(entry);
}
} catch (LdapException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} catch (CursorException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} finally {
cursor.close();
try {
connection.close();
} catch (IOException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
}
return p;
}
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, January 27, 2015 6:49 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 28/01/15 01:27, Harris, Christopher P a écrit :
> O.K. I think I remember why I stopped looking at the SearchScope. If I just change the scope to either ONELEVEL or OBJECT, the cursor comes back empty.
ONE_LEVEL applied on a terminal entry will return nothing. With OBJECT, if the entry exists, it should work.
If you can test the simplest possible code that grab an entry with OBJECT scope, and if it does not work, then it may be a bug, but then, I'd like to know about.
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 28/01/15 01:27, Harris, Christopher P a écrit :
> O.K. I think I remember why I stopped looking at the SearchScope. If I just change the scope to either ONELEVEL or OBJECT, the cursor comes back empty.
ONE_LEVEL applied on a terminal entry will return nothing. With OBJECT,
if the entry exists, it should work.
If you can test the simplest possible code that grab an entry with
OBJECT scope, and if it does not work, then it may be a bug, but then,
I'd like to know about.
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
O.K. I think I remember why I stopped looking at the SearchScope. If I just change the scope to either ONELEVEL or OBJECT, the cursor comes back empty.
Is this an AD "feature" or is this a bug in the API?
It looks like I'll need to do 1 query using the SUBTREE scope.
- Chris
-----Original Message-----
From: Harris, Christopher P
Sent: Tuesday, January 27, 2015 6:16 PM
To: api@directory.apache.org
Subject: RE: Proper use of LdapConnectionPool
Ah, crap. I forgot to look at the Scope. I've been using this code for so long for single-search queries that I took it for granted.
I'll try setting it to ONE_LEVEL to simply bask in the glory of the speedy results, but still, doing just 1 query makes a hell of a lot more sense.
Sorry, I don't know where my head was.
Thanks for steering me down the right path.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, January 27, 2015 5:43 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 27/01/15 23:07, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> "Can you tell us how you do that ? Ie, are you using a plain new connection for each thread you spawn ?"
> Sure. I can tell you how I am implementing a multi-threaded approach to read all of LDAP/AD into memory. I'll do the next best thing...paste my code at the end of my response.
>
>
> "In any case, the TimeOut is the default LDapConnection timeout (30 seconds) :"
> Yes, I noticed mention of the default timeout in your User Guide.
>
>
> "You have to set the LdapConnectionConfig timeout for all the created connections to use it. there is a setTimeout() method for that which has been added in 1.0.0-M28."
> When visiting your site while seeking to explore connection pool options, I noticed that you recently released M28 and fixed DIRAPI-217 and decided to update my pom.xml to M28 and test out the PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
>
> Oh, and your example needs to be updated to using DefaultPoolableLdapConnectionFactory instead of PoolableLdapConnectionFactory.
>
>
> "config.setTimeOut( whatever fits you );"
> Very good to know. Thank you!
>
>
> "It is the right way."
> Sweeeeeeet!
>
>
> "Side note : you may face various problems when pulling everything
> from an AD server. Typically, the AD config might not let you pull
> more than
> 1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.
>
> Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is limited. Pulling entries from the server is fast, you should be able to get tens of thousands per second with one single thread. I'm not sure how AD support concurrent searches anyway. Last, not least, it's likely that AD does not allow more than a certain number of concurrent threads to run, which might lead to contention at some point."
>
> Ah, this is why I wanted to reach out to you guys. You guys know this kind of in-depth information about LDAP and AD. So, I may adapt my code to a single-thread then. I can live with that. I need to pull about 40k-60k entries, so 10's of thousands of entries per second works for me. I may need to run the code by you then if I go with a single-threaded approach and need to check if I'm going about it in the most efficient manner.
The pb with the multi-threaded approach is that you *have* to know which entry has children, because they won't give you such an info. So you will end doing a search for every single entry you get at one level, with scope ONE_LEVEL, and most of the time, you will just get teh entry itself. That would more than double the time it takes to grab everything.
>
>
>
> And now time for some code...
>
> import java.io.IOException;
> import java.util.Iterator;
> import java.util.List;
> import java.util.Map;
> import java.util.concurrent.ConcurrentHashMap;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors; import
> java.util.concurrent.TimeUnit; import java.util.logging.Level; import
> java.util.logging.Logger;
>
> import org.apache.commons.pool.impl.GenericObjectPool;
> import org.apache.directory.api.ldap.model.cursor.CursorException;
> import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> import org.apache.directory.api.ldap.model.entry.Entry;
> import org.apache.directory.api.ldap.model.exception.LdapException;
> import org.apache.directory.api.ldap.model.message.Response;
> import org.apache.directory.api.ldap.model.message.SearchRequest;
> import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> import org.apache.directory.api.ldap.model.message.SearchResultEntry;
> import org.apache.directory.api.ldap.model.message.SearchScope;
> import org.apache.directory.api.ldap.model.name.Dn;
> import
> org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> import org.apache.directory.ldap.client.api.LdapConnection;
> import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> import org.apache.directory.ldap.client.api.LdapConnectionPool;
> import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> import
> org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFact
> ory; import
> org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionF
> actory; import org.apache.directory.ldap.client.api.SearchCursorImpl;
> import org.apache.directory.ldap.client.template.EntryMapper;
> import
> org.apache.directory.ldap.client.template.LdapConnectionTemplate;
>
> /**
> * @author Chris Harris
> *
> */
> public class LdapClient {
>
> public LdapClient() {
>
> }
>
> public Person searchLdapForCeo() {
> return this.searchLdapUsingHybridApproach(ceoQuery);
> }
>
> public Map<String, Person> buildLdapMap() {
> SearchCursor cursor = new SearchCursorImpl(null, 300000, TimeUnit.SECONDS);
> LdapConnection connection = new LdapNetworkConnection(host, port);
> connection.setTimeOut(300000);
> Entry entry = null;
>
> try {
> connection.bind(dn, pwd);
> LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, ceoQuery);
> System.out.println("Finished all Ldap Map Builder threads...");
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } catch (CursorException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } finally {
> cursor.close();
> try {
> connection.close();
> } catch (IOException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> }
> }
>
> return concurrentPersonMap;
> }
>
> private static Person recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor cursor, Entry entry, String query)
> throws CursorException {
> Person p = null;
> EntryMapper<Person> em = Person.getEntryMapper();
>
> try {
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> StringBuilder sb = new StringBuilder(query);
> sr.setFilter(sb.toString());
> sr.setScope( SearchScope.SUBTREE );
Ahhhhh !!!! STOP !!!
Ok, no need to go any further in your code.
You are doing a SUBTREE search on *every single entry* you are pulling from the base. if you have 40 000 entries, you will do something like O(
40 000! ) (factorial) searches. No wonder why you get timeout... Imagine you have such a tree :
root
A1
B1
C1
C2
B2
C3
C4
A2
B3
C5
C6
B4
C7
C8
The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
-> 14 searches)
Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
searches)
Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
searches)
Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 = 8 ...
At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have only 15 entries...
If you want to see what your algorithm is doing, just do a search using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40 000) searches, which is way less than what you are doing.
But anyway, doing a search on the root with a SUBTREE scope will be way faster, because you will do only one single search.
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Ah, crap. I forgot to look at the Scope. I've been using this code for so long for single-search queries that I took it for granted.
I'll try setting it to ONE_LEVEL to simply bask in the glory of the speedy results, but still, doing just 1 query makes a hell of a lot more sense.
Sorry, I don't know where my head was.
Thanks for steering me down the right path.
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, January 27, 2015 5:43 PM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 27/01/15 23:07, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> "Can you tell us how you do that ? Ie, are you using a plain new connection for each thread you spawn ?"
> Sure. I can tell you how I am implementing a multi-threaded approach to read all of LDAP/AD into memory. I'll do the next best thing...paste my code at the end of my response.
>
>
> "In any case, the TimeOut is the default LDapConnection timeout (30 seconds) :"
> Yes, I noticed mention of the default timeout in your User Guide.
>
>
> "You have to set the LdapConnectionConfig timeout for all the created connections to use it. there is a setTimeout() method for that which has been added in 1.0.0-M28."
> When visiting your site while seeking to explore connection pool options, I noticed that you recently released M28 and fixed DIRAPI-217 and decided to update my pom.xml to M28 and test out the PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
>
> Oh, and your example needs to be updated to using DefaultPoolableLdapConnectionFactory instead of PoolableLdapConnectionFactory.
>
>
> "config.setTimeOut( whatever fits you );"
> Very good to know. Thank you!
>
>
> "It is the right way."
> Sweeeeeeet!
>
>
> "Side note : you may face various problems when pulling everything
> from an AD server. Typically, the AD config might not let you pull
> more than
> 1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.
>
> Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is limited. Pulling entries from the server is fast, you should be able to get tens of thousands per second with one single thread. I'm not sure how AD support concurrent searches anyway. Last, not least, it's likely that AD does not allow more than a certain number of concurrent threads to run, which might lead to contention at some point."
>
> Ah, this is why I wanted to reach out to you guys. You guys know this kind of in-depth information about LDAP and AD. So, I may adapt my code to a single-thread then. I can live with that. I need to pull about 40k-60k entries, so 10's of thousands of entries per second works for me. I may need to run the code by you then if I go with a single-threaded approach and need to check if I'm going about it in the most efficient manner.
The pb with the multi-threaded approach is that you *have* to know which entry has children, because they won't give you such an info. So you will end doing a search for every single entry you get at one level, with scope ONE_LEVEL, and most of the time, you will just get teh entry itself. That would more than double the time it takes to grab everything.
>
>
>
> And now time for some code...
>
> import java.io.IOException;
> import java.util.Iterator;
> import java.util.List;
> import java.util.Map;
> import java.util.concurrent.ConcurrentHashMap;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors; import
> java.util.concurrent.TimeUnit; import java.util.logging.Level; import
> java.util.logging.Logger;
>
> import org.apache.commons.pool.impl.GenericObjectPool;
> import org.apache.directory.api.ldap.model.cursor.CursorException;
> import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> import org.apache.directory.api.ldap.model.entry.Entry;
> import org.apache.directory.api.ldap.model.exception.LdapException;
> import org.apache.directory.api.ldap.model.message.Response;
> import org.apache.directory.api.ldap.model.message.SearchRequest;
> import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> import org.apache.directory.api.ldap.model.message.SearchResultEntry;
> import org.apache.directory.api.ldap.model.message.SearchScope;
> import org.apache.directory.api.ldap.model.name.Dn;
> import
> org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> import org.apache.directory.ldap.client.api.LdapConnection;
> import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> import org.apache.directory.ldap.client.api.LdapConnectionPool;
> import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> import
> org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFact
> ory; import
> org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionF
> actory; import org.apache.directory.ldap.client.api.SearchCursorImpl;
> import org.apache.directory.ldap.client.template.EntryMapper;
> import
> org.apache.directory.ldap.client.template.LdapConnectionTemplate;
>
> /**
> * @author Chris Harris
> *
> */
> public class LdapClient {
>
> public LdapClient() {
>
> }
>
> public Person searchLdapForCeo() {
> return this.searchLdapUsingHybridApproach(ceoQuery);
> }
>
> public Map<String, Person> buildLdapMap() {
> SearchCursor cursor = new SearchCursorImpl(null, 300000, TimeUnit.SECONDS);
> LdapConnection connection = new LdapNetworkConnection(host, port);
> connection.setTimeOut(300000);
> Entry entry = null;
>
> try {
> connection.bind(dn, pwd);
> LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, ceoQuery);
> System.out.println("Finished all Ldap Map Builder threads...");
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } catch (CursorException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } finally {
> cursor.close();
> try {
> connection.close();
> } catch (IOException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> }
> }
>
> return concurrentPersonMap;
> }
>
> private static Person recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor cursor, Entry entry, String query)
> throws CursorException {
> Person p = null;
> EntryMapper<Person> em = Person.getEntryMapper();
>
> try {
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> StringBuilder sb = new StringBuilder(query);
> sr.setFilter(sb.toString());
> sr.setScope( SearchScope.SUBTREE );
Ahhhhh !!!! STOP !!!
Ok, no need to go any further in your code.
You are doing a SUBTREE search on *every single entry* you are pulling from the base. if you have 40 000 entries, you will do something like O(
40 000! ) (factorial) searches. No wonder why you get timeout... Imagine you have such a tree :
root
A1
B1
C1
C2
B2
C3
C4
A2
B3
C5
C6
B4
C7
C8
The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
-> 14 searches)
Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
searches)
Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
searches)
Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 = 8 ...
At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have only 15 entries...
If you want to see what your algorithm is doing, just do a search using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40 000) searches, which is way less than what you are doing.
But anyway, doing a search on the root with a SUBTREE scope will be way faster, because you will do only one single search.
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 27/01/15 23:07, Harris, Christopher P a écrit :
> Hi, Emmanuel.
>
> "Can you tell us how you do that ? Ie, are you using a plain new connection for each thread you spawn ?"
> Sure. I can tell you how I am implementing a multi-threaded approach to read all of LDAP/AD into memory. I'll do the next best thing...paste my code at the end of my response.
>
>
> "In any case, the TimeOut is the default LDapConnection timeout (30 seconds) :"
> Yes, I noticed mention of the default timeout in your User Guide.
>
>
> "You have to set the LdapConnectionConfig timeout for all the created connections to use it. there is a setTimeout() method for that which has been added in 1.0.0-M28."
> When visiting your site while seeking to explore connection pool options, I noticed that you recently released M28 and fixed DIRAPI-217 and decided to update my pom.xml to M28 and test out the PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
>
> Oh, and your example needs to be updated to using DefaultPoolableLdapConnectionFactory instead of PoolableLdapConnectionFactory.
>
>
> "config.setTimeOut( whatever fits you );"
> Very good to know. Thank you!
>
>
> "It is the right way."
> Sweeeeeeet!
>
>
> "Side note : you may face various problems when pulling everything from an AD server. Typically, the AD config might not let you pull more than
> 1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.
>
> Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is limited. Pulling entries from the server is fast, you should be able to get tens of thousands per second with one single thread. I'm not sure how AD support concurrent searches anyway. Last, not least, it's likely that AD does not allow more than a certain number of concurrent threads to run, which might lead to contention at some point."
>
> Ah, this is why I wanted to reach out to you guys. You guys know this kind of in-depth information about LDAP and AD. So, I may adapt my code to a single-thread then. I can live with that. I need to pull about 40k-60k entries, so 10's of thousands of entries per second works for me. I may need to run the code by you then if I go with a single-threaded approach and need to check if I'm going about it in the most efficient manner.
The pb with the multi-threaded approach is that you *have* to know which
entry has children, because they won't give you such an info. So you
will end doing a search for every single entry you get at one level,
with scope ONE_LEVEL, and most of the time, you will just get teh entry
itself. That would more than double the time it takes to grab everything.
>
>
>
> And now time for some code...
>
> import java.io.IOException;
> import java.util.Iterator;
> import java.util.List;
> import java.util.Map;
> import java.util.concurrent.ConcurrentHashMap;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.concurrent.TimeUnit;
> import java.util.logging.Level;
> import java.util.logging.Logger;
>
> import org.apache.commons.pool.impl.GenericObjectPool;
> import org.apache.directory.api.ldap.model.cursor.CursorException;
> import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> import org.apache.directory.api.ldap.model.entry.Entry;
> import org.apache.directory.api.ldap.model.exception.LdapException;
> import org.apache.directory.api.ldap.model.message.Response;
> import org.apache.directory.api.ldap.model.message.SearchRequest;
> import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> import org.apache.directory.api.ldap.model.message.SearchResultEntry;
> import org.apache.directory.api.ldap.model.message.SearchScope;
> import org.apache.directory.api.ldap.model.name.Dn;
> import org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> import org.apache.directory.ldap.client.api.LdapConnection;
> import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> import org.apache.directory.ldap.client.api.LdapConnectionPool;
> import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> import org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFactory;
> import org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory;
> import org.apache.directory.ldap.client.api.SearchCursorImpl;
> import org.apache.directory.ldap.client.template.EntryMapper;
> import org.apache.directory.ldap.client.template.LdapConnectionTemplate;
>
> /**
> * @author Chris Harris
> *
> */
> public class LdapClient {
>
> public LdapClient() {
>
> }
>
> public Person searchLdapForCeo() {
> return this.searchLdapUsingHybridApproach(ceoQuery);
> }
>
> public Map<String, Person> buildLdapMap() {
> SearchCursor cursor = new SearchCursorImpl(null, 300000, TimeUnit.SECONDS);
> LdapConnection connection = new LdapNetworkConnection(host, port);
> connection.setTimeOut(300000);
> Entry entry = null;
>
> try {
> connection.bind(dn, pwd);
> LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, ceoQuery);
> System.out.println("Finished all Ldap Map Builder threads...");
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } catch (CursorException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } finally {
> cursor.close();
> try {
> connection.close();
> } catch (IOException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> }
> }
>
> return concurrentPersonMap;
> }
>
> private static Person recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor cursor, Entry entry, String query)
> throws CursorException {
> Person p = null;
> EntryMapper<Person> em = Person.getEntryMapper();
>
> try {
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> StringBuilder sb = new StringBuilder(query);
> sr.setFilter(sb.toString());
> sr.setScope( SearchScope.SUBTREE );
Ahhhhh !!!! STOP !!!
Ok, no need to go any further in your code.
You are doing a SUBTREE search on *every single entry* you are pulling
from the base. if you have 40 000 entries, you will do something like O(
40 000! ) (factorial) searches. No wonder why you get timeout... Imagine
you have such a tree :
root
A1
B1
C1
C2
B2
C3
C4
A2
B3
C5
C6
B4
C7
C8
The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
-> 14 searches)
Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
searches)
Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
searches)
Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 = 8
...
At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have
only 15 entries...
If you want to see what your algorithm is doing, just do a search using
a SearchScope.ONE_LEVEL instead. You will only do somehow O(40 000)
searches, which is way less than what you are doing.
But anyway, doing a search on the root with a SUBTREE scope will be way
faster, because you will do only one single search.
RE: Proper use of LdapConnectionPool
Posted by "Harris, Christopher P" <ch...@baxter.com>.
Hi, Emmanuel.
"Can you tell us how you do that ? Ie, are you using a plain new connection for each thread you spawn ?"
Sure. I can tell you how I am implementing a multi-threaded approach to read all of LDAP/AD into memory. I'll do the next best thing...paste my code at the end of my response.
"In any case, the TimeOut is the default LDapConnection timeout (30 seconds) :"
Yes, I noticed mention of the default timeout in your User Guide.
"You have to set the LdapConnectionConfig timeout for all the created connections to use it. there is a setTimeout() method for that which has been added in 1.0.0-M28."
When visiting your site while seeking to explore connection pool options, I noticed that you recently released M28 and fixed DIRAPI-217 and decided to update my pom.xml to M28 and test out the PoolableLdapConnectionFactory. Great job, btw. Keep up the good work!
Oh, and your example needs to be updated to using DefaultPoolableLdapConnectionFactory instead of PoolableLdapConnectionFactory.
"config.setTimeOut( whatever fits you );"
Very good to know. Thank you!
"It is the right way."
Sweeeeeeet!
"Side note : you may face various problems when pulling everything from an AD server. Typically, the AD config might not let you pull more than
1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.
Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is limited. Pulling entries from the server is fast, you should be able to get tens of thousands per second with one single thread. I'm not sure how AD support concurrent searches anyway. Last, not least, it's likely that AD does not allow more than a certain number of concurrent threads to run, which might lead to contention at some point."
Ah, this is why I wanted to reach out to you guys. You guys know this kind of in-depth information about LDAP and AD. So, I may adapt my code to a single-thread then. I can live with that. I need to pull about 40k-60k entries, so 10's of thousands of entries per second works for me. I may need to run the code by you then if I go with a single-threaded approach and need to check if I'm going about it in the most efficient manner.
And now time for some code...
import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.commons.pool.impl.GenericObjectPool;
import org.apache.directory.api.ldap.model.cursor.CursorException;
import org.apache.directory.api.ldap.model.cursor.SearchCursor;
import org.apache.directory.api.ldap.model.entry.Entry;
import org.apache.directory.api.ldap.model.exception.LdapException;
import org.apache.directory.api.ldap.model.message.Response;
import org.apache.directory.api.ldap.model.message.SearchRequest;
import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
import org.apache.directory.api.ldap.model.message.SearchResultEntry;
import org.apache.directory.api.ldap.model.message.SearchScope;
import org.apache.directory.api.ldap.model.name.Dn;
import org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
import org.apache.directory.ldap.client.api.LdapConnection;
import org.apache.directory.ldap.client.api.LdapConnectionConfig;
import org.apache.directory.ldap.client.api.LdapConnectionPool;
import org.apache.directory.ldap.client.api.LdapNetworkConnection;
import org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFactory;
import org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory;
import org.apache.directory.ldap.client.api.SearchCursorImpl;
import org.apache.directory.ldap.client.template.EntryMapper;
import org.apache.directory.ldap.client.template.LdapConnectionTemplate;
/**
* @author Chris Harris
*
*/
public class LdapClient {
public LdapClient() {
}
public Person searchLdapForCeo() {
return this.searchLdapUsingHybridApproach(ceoQuery);
}
public Map<String, Person> buildLdapMap() {
SearchCursor cursor = new SearchCursorImpl(null, 300000, TimeUnit.SECONDS);
LdapConnection connection = new LdapNetworkConnection(host, port);
connection.setTimeOut(300000);
Entry entry = null;
try {
connection.bind(dn, pwd);
LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, ceoQuery);
System.out.println("Finished all Ldap Map Builder threads...");
} catch (LdapException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} catch (CursorException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} finally {
cursor.close();
try {
connection.close();
} catch (IOException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
}
return concurrentPersonMap;
}
private static Person recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor cursor, Entry entry, String query)
throws CursorException {
Person p = null;
EntryMapper<Person> em = Person.getEntryMapper();
try {
SearchRequest sr = new SearchRequestImpl();
sr.setBase(new Dn(searchBase));
StringBuilder sb = new StringBuilder(query);
sr.setFilter(sb.toString());
sr.setScope( SearchScope.SUBTREE );
cursor = connection.search(sr);
Response response;
while (cursor.next() && cursor.isEntry()) {
response = cursor.get();
System.out.println(((SearchResultEntry)response).getEntry());
entry = cursor.getEntry();
// Catches entries that are disabled via the userAccountControl being set to anything other than 512
if ("512".equals(entry.get( "userAccountControl" ).getString())) {
p = em.map(entry);
concurrentPersonMap.put(p.getDistinguishedName(), p);
if (p.getDirectReports() != null && p.getDirectReports().size() != 0) {
ExecutorService executor = Executors.newFixedThreadPool(p.getDirectReports().size());
for (String directReportDistinguishedName : p.getDirectReports()) {
Runnable worker = new LdapPersonRetrieverRunnable(connection, cursor, entry,
LdapDistinguishedNameUtil.buildDistinguishedNameLdapQuery(directReportDistinguishedName));
executor.execute(worker);
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
// Wait until all threads are finish
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
}
} else {
// Remove the Direct Report from the Manager's list of direct reports
if (entry.get( "manager" ).getString() != null) {
Iterator<String> it = concurrentPersonMap.get(entry.get( "manager" ).getString()).getDirectReports().iterator();
while (it.hasNext()) {
if ((entry.getDn().getName()).equals(it.next())) {
it.remove();
}
}
}
}
}
} catch (InterruptedException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
} catch (LdapException ex) {
System.out.println("LDAP API hung on this distinguishedName query: " + query);
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
return p;
}
private static class LdapPersonRetrieverRunnable implements Runnable {
private LdapConnection connection;
private SearchCursor cursor;
private Entry entry;
private String query;
public LdapPersonRetrieverRunnable(LdapConnection connection, SearchCursor cursor, Entry entry, String query) {
this.connection = connection;
this.cursor = cursor;
this.entry = entry;
this.query = query;
}
@Override
public void run() {
try {
LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, query);
} catch (CursorException ex) {
Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* @author Chris Harris
*
*/
public class LdapDistinguishedNameUtil {
public static String buildDistinguishedNameLdapQuery(String distinguishedName) {
StringBuilder sb = new StringBuilder();
sb.append("(&(objectClass=person)(distinguishedName=")
.append(replaceFilterSpecialCharactersWithHexRepresentation(distinguishedName))
.append("))");
return sb.toString();
}
public static String replaceFilterSpecialCharactersWithHexRepresentation(String distinguishedName) {
Map<String, String> replacements = new HashMap<String, String>() {
private static final long serialVersionUID = 5562696261585474518L;
{
put("*", "\\\\2A");
put("(", "\\\\28");
put(")", "\\\\29");
put("\\", "\\\\5C");
}};
String regexp = "\\(|\\)|\\*|\\\\";
StringBuffer sb = new StringBuffer();
Pattern p = Pattern.compile(regexp);
Matcher m = p.matcher(distinguishedName);
while (m.find()) {
m.appendReplacement(sb, replacements.get(m.group()));
}
m.appendTail(sb);
return sb.toString();
}
public static boolean containsIgnoreCase(String src, String what) {
final int length = what.length();
if (length == 0)
return true; // Empty string is contained
final char firstLo = Character.toLowerCase(what.charAt(0));
final char firstUp = Character.toUpperCase(what.charAt(0));
for (int i = src.length() - length; i >= 0; i--) {
// Quick check before calling the more expensive regionMatches() method:
final char ch = src.charAt(i);
if (ch != firstLo && ch != firstUp) {
continue;
}
if (src.regionMatches(true, i, what, 0, length)) {
return true;
}
}
return false;
}
}
import java.util.ArrayList;
import java.util.Iterator;
import org.apache.directory.api.ldap.model.entry.Attribute;
import org.apache.directory.api.ldap.model.entry.Entry;
import org.apache.directory.api.ldap.model.entry.Value;
import org.apache.directory.api.ldap.model.exception.LdapException;
import org.apache.directory.ldap.client.template.EntryMapper;
/**
* @author Chris Harris
*
*/
public class Person {
protected Person() {
}
private static final EntryMapper<Person> personEntryMapper =
new EntryMapper<Person>() {
@Override
public Person map( Entry entry ) throws LdapException {
try {
return new Person.Builder()
.setDistinguishedName(entry.getDn().getName())
.setFirstName(entry.get( "givenName" ).getString())
.setLastName(entry.get( "sn" ).getString())
.setTitle(entry.get( "title" ).getString())
.setDepartment(entry.get( "department" ).getString())
.setDivision(entry.get( "extensionAttribute14" ).getString())
.setCity(entry.get( "l" ).getString())
.setCountry(entry.get( "co" ).getString())
.setEmail((entry.get( "mail" ) != null) ? entry.get( "mail" ).getString() : "")
.setDeskLocation(entry.get( "postOfficeBox" ).getString())
.setOfficePhone((entry.get( "telephoneNumber" ) != null) ? entry.get( "telephoneNumber" ).getString() : "")
.setDirectReports(entry.get( "directreports" ))
.build();
} catch ( Exception e ) {
System.out.println( "GOTCHA: " + e.getMessage() );
e.printStackTrace();
try {
throw( e );
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
return null;
}
};
public static EntryMapper<Person> getEntryMapper() {
return personEntryMapper;
}
public static class Builder {
private Person person;
public Builder() {
this.person = new Person();
}
public Builder setDistinguishedName(String distinguishedName) {
this.person.distinguishedName = distinguishedName;
return this;
}
public Builder setFirstName(String firstName) {
this.person.firstName = firstName;
return this;
}
public Builder setLastName(String lastName) {
this.person.lastName = lastName;
return this;
}
public Builder setDirectReports(Attribute directReports) {
if (directReports == null) {
this.person.directReports = null;
} else {
Iterator<Value<?>> it = directReports.iterator();
ArrayList<String> dirReports = new ArrayList<String>();
while (it.hasNext()) {
Value<?> val = it.next();
if (!LdapDistinguishedNameUtil.containsIgnoreCase(val.getString(), "Group Mailboxes") ||
!LdapDistinguishedNameUtil.containsIgnoreCase(val.getString(), "Disabled Accounts")) {
dirReports.add(val.getString());
}
}
this.person.directReports = dirReports;
}
return this;
}
public Builder setTitle(String title) {
this.person.title = title;
return this;
}
public Builder setDepartment(String department) {
this.person.department = department;
return this;
}
public Builder setDivision(String division) {
this.person.division = division;
return this;
}
public Builder setCity(String city) {
this.person.city = city;
return this;
}
public Builder setCountry(String country) {
this.person.country = country;
return this;
}
public Builder setEmail(String email) {
this.person.email = email;
return this;
}
public Builder setDeskLocation(String deskLocation) {
this.person.deskLocation = deskLocation;
return this;
}
public Builder setOfficePhone(String officePhone) {
this.person.officePhone = officePhone;
return this;
}
public Person build() {
return this.person;
}
}
private String distinguishedName;
private String firstName;
private String lastName;
private ArrayList<String> directReports;
private String title;
private String department;
private String division;
private String city;
private String country;
private String email;
private String deskLocation;
private String officePhone;
public String getDistinguishedName() {
return distinguishedName;
}
public void setDistinguishedName(String distinguishedName) {
this.distinguishedName = distinguishedName;
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public ArrayList<String> getDirectReports() {
return directReports;
}
public void setDirectReports(ArrayList<String> directReports) {
this.directReports = directReports;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getDepartment() {
return department;
}
public void setDepartment(String department) {
this.department = department;
}
public String getDivision() {
return division;
}
public void setDivision(String division) {
this.division = division;
}
public String getCity() {
return city;
}
public void setCity(String city) {
this.city = city;
}
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public String getDeskLocation() {
return deskLocation;
}
public void setDeskLocation(String deskLocation) {
this.deskLocation = deskLocation;
}
public String getOfficePhone() {
return officePhone;
}
public void setOfficePhone(String officePhone) {
this.officePhone = officePhone;
}
}
...and finally, the relevant Junit 4.12 code snippet:
@Test
public void testLdapTreeBuilder() {
long start = System.currentTimeMillis();
Map<String, Person> ldapPersonMap = ldapClient.buildLdapMap();
assertNotNull(ldapPersonMap);
long end = System.currentTimeMillis();
System.out.println("Building the Ldap map took " + Long.toString(end - start) + " MilliSeconds");
}
- Chris
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, January 27, 2015 2:23 AM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool
Le 27/01/15 06:11, Harris, Christopher P a écrit :
> Hi,
>
> I'm running into TimeOut issues when implementing a multi-threaded approach to read all of LDAP/AD into memory, starting with the CEO and trickling down.
Can you tell us how you do that ? Ie, are you using a plain new connection for each thread you spawn ?
In any case, the TimeOut is the default LDapConnection timeout (30
seconds) :
/** The timeout used for response we are waiting for */
private long timeout = LdapConnectionConfig.DEFAULT_TIMEOUT;
/** The default timeout for operation : 30 seconds */
public static final long DEFAULT_TIMEOUT = 30000L;
That means the server is *not* responding for more than 30 seconds.
> I haven't been using the LdapConnectionPool, so I thought that I'd give that a try to see if it solves my TimeOut issues. I'm setting the timeout for my cursor and connection to 300,000 milliseconds, however the timeout consistently occurs around 30 seconds I've noticed.
You have to set the LdapConnectionConfig timeout for all the created connections to use it. there is a setTimeout() method for that which has been added in 1.0.0-M28.
>
> Anyway, I read the docs concerning implementing an LdapConnectionPool, but the example doesn't show you how to use the connection pool.
>
> Does the following method use the LdapConnectionPool the correct way?
>
> public Person searchLdapUsingConnectionPool() {
> SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
> LdapConnectionPool pool = null;
> LdapConnection connection = null;
> Person p = null;
> try {
> LdapConnectionConfig config = new LdapConnectionConfig();
> config.setLdapHost( host );
> config.setLdapPort( port );
> config.setName( dn );
> config.setCredentials( pwd );
> DefaultPoolableLdapConnectionFactory factory = new DefaultPoolableLdapConnectionFactory( config );
> pool = new LdapConnectionPool( factory );
> pool.setTestOnBorrow( true );
> connection = pool.getConnection();
It's all good. You can now change the config timeout :
config.setTimeOut( whatever fits you );
>
> Entry entry = null;
>
>
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> StringBuilder sb = new StringBuilder(ceoQuery);
> sr.setFilter(sb.toString());
> sr.setScope( SearchScope.SUBTREE );
> cursor = connection.search(sr);
> Response response;
>
> while (cursor.next() && cursor.isEntry()) {
> response = cursor.get();
> System.out.println(((SearchResultEntry)response).getEntry());
> entry = cursor.getEntry();
> EntryMapper<Person> em = Person.getEntryMapper();
> p = em.map(entry);
> }
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } catch (CursorException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } finally {
> cursor.close();
> try {
> pool.releaseConnection(connection);
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> }
> }
>
> return p;
> }
>
> It seems like I just need to grab a connection via pool.getConnection(), but I don't know for sure if I'm using it the recommended way.
It is the right way.
Side note : you may face various problems when pulling everything from an AD server. Typically, the AD config might not let you pull more than
1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.
Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is limited. Pulling entries from the server is fast, you should be able to get tens of thousands per second with one single thread. I'm not sure how AD support concurrent searches anyway. Last, not least, it's likely that AD does not allow more than a certain number of concurrent threads to run, which might lead to contention at some point.
Hope it helps.
Emmanuel
The information transmitted is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or legally privileged material. Delivery of this message to any person other than the intended recipient(s) is not intended in any way to waive privilege or confidentiality. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by entities other than the intended recipient is prohibited. If you receive this in error, please contact the sender and delete the material from any computer.
For Translation:
http://www.baxter.com/email_disclaimer
Re: Proper use of LdapConnectionPool
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 27/01/15 06:11, Harris, Christopher P a écrit :
> Hi,
>
> I'm running into TimeOut issues when implementing a multi-threaded approach to read all of LDAP/AD into memory, starting with the CEO and trickling down.
Can you tell us how you do that ? Ie, are you using a plain new
connection for each thread you spawn ?
In any case, the TimeOut is the default LDapConnection timeout (30
seconds) :
/** The timeout used for response we are waiting for */
private long timeout = LdapConnectionConfig.DEFAULT_TIMEOUT;
/** The default timeout for operation : 30 seconds */
public static final long DEFAULT_TIMEOUT = 30000L;
That means the server is *not* responding for more than 30 seconds.
> I haven't been using the LdapConnectionPool, so I thought that I'd give that a try to see if it solves my TimeOut issues. I'm setting the timeout for my cursor and connection to 300,000 milliseconds, however the timeout consistently occurs around 30 seconds I've noticed.
You have to set the LdapConnectionConfig timeout for all the created
connections to use it. there is a setTimeout() method for that which has
been added in 1.0.0-M28.
>
> Anyway, I read the docs concerning implementing an LdapConnectionPool, but the example doesn't show you how to use the connection pool.
>
> Does the following method use the LdapConnectionPool the correct way?
>
> public Person searchLdapUsingConnectionPool() {
> SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
> LdapConnectionPool pool = null;
> LdapConnection connection = null;
> Person p = null;
> try {
> LdapConnectionConfig config = new LdapConnectionConfig();
> config.setLdapHost( host );
> config.setLdapPort( port );
> config.setName( dn );
> config.setCredentials( pwd );
> DefaultPoolableLdapConnectionFactory factory = new DefaultPoolableLdapConnectionFactory( config );
> pool = new LdapConnectionPool( factory );
> pool.setTestOnBorrow( true );
> connection = pool.getConnection();
It's all good. You can now change the config timeout :
config.setTimeOut( whatever fits you );
>
> Entry entry = null;
>
>
> SearchRequest sr = new SearchRequestImpl();
> sr.setBase(new Dn(searchBase));
> StringBuilder sb = new StringBuilder(ceoQuery);
> sr.setFilter(sb.toString());
> sr.setScope( SearchScope.SUBTREE );
> cursor = connection.search(sr);
> Response response;
>
> while (cursor.next() && cursor.isEntry()) {
> response = cursor.get();
> System.out.println(((SearchResultEntry)response).getEntry());
> entry = cursor.getEntry();
> EntryMapper<Person> em = Person.getEntryMapper();
> p = em.map(entry);
> }
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } catch (CursorException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> } finally {
> cursor.close();
> try {
> pool.releaseConnection(connection);
> } catch (LdapException ex) {
> Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
> }
> }
>
> return p;
> }
>
> It seems like I just need to grab a connection via pool.getConnection(), but I don't know for sure if I'm using it the recommended way.
It is the right way.
Side note : you may face various problems when pulling everything from
an AD server. Typically, the AD config might not let you pull more than
1000 entries, as there is a hard limit you need to change on AD if you
want to get more entries.
Otherwise, the approach - ie, using multiple threads - might seems good,
but the benefit is limited. Pulling entries from the server is fast, you
should be able to get tens of thousands per second with one single
thread. I'm not sure how AD support concurrent searches anyway. Last,
not least, it's likely that AD does not allow more than a certain number
of concurrent threads to run, which might lead to contention at some point.
Hope it helps.
Emmanuel