You are viewing a plain text version of this content. The canonical link for it is here.
Posted to api@directory.apache.org by jitesh dundas <jb...@gmail.com> on 2013/10/16 16:21:54 UTC
Need Help with the LDAP API for large records
Hi All,
I am a new user who is implementing Apache LDAP on one of our company's
servers. The records in the LDAP ( entries) come from a text file ( LDAP
dump) having 600000 records in it.
I have written a utility to insert it buyt it keeps failing with these
errors:-
[10:16:25] ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind failed
: timeout occurred
[10:16:25] ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection] - The response
queue has been emptied, no response was found.
org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
occurred
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
at
org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
at
org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
at Main.getConnection(Main.java:38)
at Main.main(Main.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Here is the code to add it:-
public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) throws
LdapException, IOException {
//Sonar Ldap Connection Details
Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
"secret");
List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
for(int i=6000; i<webphonePersons.size() ; i++){
LdapConnection connection = client.getConnection();
connection.setTimeOut(1000000);
try {
//create the new entries to add.
Entry sonarLdapEntry = new
DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry goes
here.
sonarLdapEntry.add("objectClass","top");
sonarLdapEntry.add("objectClass","inetOrgPerson");
sonarLdapEntry.add("objectClass","person");
sonarLdapEntry.add("objectClass","organizationalPerson");
sonarLdapEntry.add("cn", webphonePersons.get(i).getUID());
sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
webphonePersons.get(i).getLastName());
sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
connection.add(sonarLdapEntry);
LOGGER.info("Sonar Test LDAP@Added User
sonarLdapEntry=>"+sonarLdapEntry.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
//e.printStackTrace();
System.out.println("UID FAILED => "+
webphonePersons.get(i).getAttUID());
}
//closing connections after every 1000 records added and
then opened again.
//to prevent memory issues.
if (i%1000 == 0){
client.closeConnection(connection);
connection = client.getConnection();
}
}
}
Any ideas on how to resolve it? We have the Java memory options set to
-Xms1024m -Xmx1024m
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
Hi Kiran,
Well it seems that this is working much better. Removing the extra
connections and giving a sleep condition as shown in my previous email has
not given the error till now.
It is still running but I guess it should be done by tonight.
Will confirm when all is done with you all on this list so that others can
get help later if needed.
Special Thanks to Kiran for his prompt reply. Appreciate it :)
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 11:33 AM, jitesh dundas <jb...@gmail.com> wrote:
> Hi Kiran,
>
> Thanks again for the reply. Appreciate it.
>
> We are using the latest ApacheDS 2.0.0 version. I modified the method to
> look like this:-
>
> LdapConnection connection = client.getConnection();
> connection.setTimeOut(1000000000);
>
> Date date = new Date();
> String dateStr = "Started="+date.getDate() + "-" + date.getMonth() +
> "-" + date.getYear() ;
> System.out.println("START@"+ dateStr );
>
>
> List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
> System.out.println("Found Users" + webphonePersons.size());
> for(int i=18000; i<webphonePersons.size() ; i++){
>
>
> try {
> //create the new entries to add.
> Entry sonarLdapEntry = new DefaultEntry("cn="+
> webphonePersons.get(i).getAttUID() +",ou=people,O=ATT,C=US");//dn of the
> entry goes here.
>
> sonarLdapEntry.add("objectClass","top");
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> sonarLdapEntry.add("objectClass","person");
> sonarLdapEntry.add("objectClass","organizationalPerson");
> sonarLdapEntry.add("cn",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> webphonePersons.get(i).getLastName());
> sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> connection.add(sonarLdapEntry);
> Thread.sleep(200);
> } catch (Exception e) {
> //e.printStackTrace();
> System.out.println("ATTUID FAILED => "+
> webphonePersons.get(i).getAttUID());
>
> }
> if (i%1000 == 0 || webphonePersons.size() == i){
> //client.closeConnection(connection);
>
> System.out.println("Completed => "+ i);
> Thread.sleep(4000);
> //connection = client.getConnection();
> }//end if condition
> }//end for loop
>
> Can you please tell me what is wrong here? I would love to make it better.
>
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 11:26 AM, Kiran Ayyagari <ka...@apache.org>wrote:
>
>> On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jb...@gmail.com>
>> wrote:
>>
>> > HI Kiran,
>> >
>> > Thank you for your reply. You are correct in the sense that we close the
>> > connection after every 1000 entries that are completed in the for loop
>> and
>> > then open the connection again.
>> >
>> > *if (i%1000 == 0 || webphonePersons.size() == i){*
>> > client.closeConnection(connection);
>> > System.out.println("Completed => "+ i);
>> > Thread.sleep(4000);
>> > connection = client.getConnection();
>> > }//end if condition
>> >
>> > We are also putting Thread.sleep() to wait and give some time for the
>> > connection to start/stop. Also, we have added the same sleep condition
>> of
>> > 400 milliseconds after every entry that is added( to prevent overlaps).
>> >
>> > However, based on your response, I am making the change to remove this
>> > logic and keep one connection only throughout the loop. Hopefully it
>> will
>> > work.
>> >
>> > not that your method is wrong, but one connection is enough here
>> otoh, which version of ApacheDS you are using?
>>
>> >
>> > Thanks & Regards,
>> > Jitesh Dundas
>> >
>> > Phone - 732-357-6292
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
>> > >wrote:
>> >
>> > > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Kiran,
>> > > >
>> > > > Thank you for your reply. My apologies for not being able to send
>> the
>> > > full
>> > > > java file( As you can see that I am already doing that):-
>> > > >
>> > > > public Main(String host, int port, String userName, String
>> password) {
>> > > > this.host = host;
>> > > > this.port = port;
>> > > > this.userName = userName;
>> > > > this.password = password;
>> > > > }
>> > > >
>> > > > public LdapConnection getConnection() throws LdapException,
>> > > IOException {
>> > > > LdapConnection connection = new LdapNetworkConnection(host,
>> port);
>> > > > connection.bind("uid=" + userName + ",ou=system", password);
>> > > > return connection;
>> > > > }
>> > > >
>> > > > Also, I see from the updated logs now that it traverses 3000 records
>> > and
>> > > > then it starts giving these error messages. Any ideas?
>> > > >
>> > > > Maybe it is the way in which I am making the entries.
>> > > >
>> > > > if entry creation is wrong you will get a different error (instead
>> of
>> > > bind
>> > > error)
>> > > are you opening a new connection for adding each entry? it appears so
>> to
>> > me
>> > > after seeing the method public static void
>> > addEntryIntoSonarTestLdap(Entry
>> > > labLdapEntry)
>> > >
>> > > > Thanks & Regards,
>> > > > Jitesh Dundas
>> > > >
>> > > > Phone - 732-357-6292
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <
>> kayyagari@apache.org
>> > > > >wrote:
>> > > >
>> > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <
>> jbdundas@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > I am a new user who is implementing Apache LDAP on one of our
>> > > > company's
>> > > > > > servers. The records in the LDAP ( entries) come from a text
>> file (
>> > > > LDAP
>> > > > > > dump) having 600000 records in it.
>> > > > > >
>> > > > > > I have written a utility to insert it buyt it keeps failing with
>> > > these
>> > > > > > errors:-
>> > > > > >
>> > > > > > [10:16:25] ERROR
>> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
>> Bind
>> > > > > failed
>> > > > > > : timeout occurred
>> > > > > > [10:16:25] ERROR
>> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
>> The
>> > > > > response
>> > > > > > queue has been emptied, no response was found.
>> > > > > > org.apache.directory.api.ldap.model.exception.LdapException:
>> > TimeOut
>> > > > > > occurred
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
>> > > > > > at Main.getConnection(Main.java:38)
>> > > > > > at Main.main(Main.java:130)
>> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
>> Source)
>> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>> > > Source)
>> > > > > > at java.lang.reflect.Method.invoke(Unknown Source)
>> > > > > > at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
>> > > > > >
>> > > > > >
>> > > > > > Here is the code to add it:-
>> > > > > >
>> > > > > > public static void addEntryIntoSonarTestLdap(Entry
>> labLdapEntry)
>> > > > throws
>> > > > > > LdapException, IOException {
>> > > > > > //Sonar Ldap Connection Details
>> > > > > > Main client = new Main("altv0018.aldc.att.com", 10389,
>> > > "admin",
>> > > > > > "secret");
>> > > > > >
>> > > > > replace "admin" with "uid=admin,ou=system" and try
>> > > > >
>> > > > > >
>> > > > > > List<WebphonePerson> webphonePersons =
>> > > > > WebphoneFileReader.readFile();
>> > > > > > for(int i=6000; i<webphonePersons.size() ; i++){
>> > > > > > LdapConnection connection = client.getConnection();
>> > > > > > connection.setTimeOut(1000000);
>> > > > > > try {
>> > > > > >
>> > > > > >
>> > > > > > //create the new entries to add.
>> > > > > > Entry sonarLdapEntry = new
>> > > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
>> > > entry
>> > > > > goes
>> > > > > > here.
>> > > > > > sonarLdapEntry.add("objectClass","top");
>> > > > > >
>> sonarLdapEntry.add("objectClass","inetOrgPerson");
>> > > > > > sonarLdapEntry.add("objectClass","person");
>> > > > > >
>> > > > sonarLdapEntry.add("objectClass","organizationalPerson");
>> > > > > > sonarLdapEntry.add("cn",
>> > > > > webphonePersons.get(i).getUID());
>> > > > > >
>> > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() +
>> "
>> > " +
>> > > > > > webphonePersons.get(i).getLastName());
>> > > > > >
>> > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>> > > > > >
>> > > > > >
>> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
>> > > > > > connection.add(sonarLdapEntry);
>> > > > > > LOGGER.info("Sonar Test LDAP@Added User
>> > > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
>> > > > > > } catch (Exception e) {
>> > > > > > // TODO Auto-generated catch block
>> > > > > > //e.printStackTrace();
>> > > > > > System.out.println("UID FAILED => "+
>> > > > > > webphonePersons.get(i).getAttUID());
>> > > > > > }
>> > > > > > //closing connections after every 1000 records
>> > > added
>> > > > > and
>> > > > > > then opened again.
>> > > > > > //to prevent memory issues.
>> > > > > > if (i%1000 == 0){
>> > > > > > client.closeConnection(connection);
>> > > > > > connection = client.getConnection();
>> > > > > > }
>> > > > > > }
>> > > > > >
>> > > > > > }
>> > > > > >
>> > > > > > Any ideas on how to resolve it? We have the Java memory options
>> set
>> > > to
>> > > > > > -Xms1024m -Xmx1024m
>> > > > > >
>> > > > > > Thanks & Regards,
>> > > > > > Jitesh Dundas
>> > > > > >
>> > > > > > Phone - 732-357-6292
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Kiran Ayyagari
>> > > > > http://keydap.com
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kiran Ayyagari
>> > > http://keydap.com
>> > >
>> >
>>
>>
>>
>> --
>> Kiran Ayyagari
>> http://keydap.com
>>
>
>
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
Hi Kiran,
Thanks again for the reply. Appreciate it.
We are using the latest ApacheDS 2.0.0 version. I modified the method to
look like this:-
LdapConnection connection = client.getConnection();
connection.setTimeOut(1000000000);
Date date = new Date();
String dateStr = "Started="+date.getDate() + "-" + date.getMonth() +
"-" + date.getYear() ;
System.out.println("START@"+ dateStr );
List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
System.out.println("Found Users" + webphonePersons.size());
for(int i=18000; i<webphonePersons.size() ; i++){
try {
//create the new entries to add.
Entry sonarLdapEntry = new DefaultEntry("cn="+
webphonePersons.get(i).getAttUID() +",ou=people,O=ATT,C=US");//dn of the
entry goes here.
sonarLdapEntry.add("objectClass","top");
sonarLdapEntry.add("objectClass","inetOrgPerson");
sonarLdapEntry.add("objectClass","person");
sonarLdapEntry.add("objectClass","organizationalPerson");
sonarLdapEntry.add("cn",webphonePersons.get(i).getUID());
sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
webphonePersons.get(i).getLastName());
sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
connection.add(sonarLdapEntry);
Thread.sleep(200);
} catch (Exception e) {
//e.printStackTrace();
System.out.println("ATTUID FAILED => "+
webphonePersons.get(i).getAttUID());
}
if (i%1000 == 0 || webphonePersons.size() == i){
//client.closeConnection(connection);
System.out.println("Completed => "+ i);
Thread.sleep(4000);
//connection = client.getConnection();
}//end if condition
}//end for loop
Can you please tell me what is wrong here? I would love to make it better.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 11:26 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > HI Kiran,
> >
> > Thank you for your reply. You are correct in the sense that we close the
> > connection after every 1000 entries that are completed in the for loop
> and
> > then open the connection again.
> >
> > *if (i%1000 == 0 || webphonePersons.size() == i){*
> > client.closeConnection(connection);
> > System.out.println("Completed => "+ i);
> > Thread.sleep(4000);
> > connection = client.getConnection();
> > }//end if condition
> >
> > We are also putting Thread.sleep() to wait and give some time for the
> > connection to start/stop. Also, we have added the same sleep condition of
> > 400 milliseconds after every entry that is added( to prevent overlaps).
> >
> > However, based on your response, I am making the change to remove this
> > logic and keep one connection only throughout the loop. Hopefully it will
> > work.
> >
> > not that your method is wrong, but one connection is enough here
> otoh, which version of ApacheDS you are using?
>
> >
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
> >
> >
> >
> >
> > On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com>
> > wrote:
> > >
> > > > Hi Kiran,
> > > >
> > > > Thank you for your reply. My apologies for not being able to send the
> > > full
> > > > java file( As you can see that I am already doing that):-
> > > >
> > > > public Main(String host, int port, String userName, String
> password) {
> > > > this.host = host;
> > > > this.port = port;
> > > > this.userName = userName;
> > > > this.password = password;
> > > > }
> > > >
> > > > public LdapConnection getConnection() throws LdapException,
> > > IOException {
> > > > LdapConnection connection = new LdapNetworkConnection(host,
> port);
> > > > connection.bind("uid=" + userName + ",ou=system", password);
> > > > return connection;
> > > > }
> > > >
> > > > Also, I see from the updated logs now that it traverses 3000 records
> > and
> > > > then it starts giving these error messages. Any ideas?
> > > >
> > > > Maybe it is the way in which I am making the entries.
> > > >
> > > > if entry creation is wrong you will get a different error (instead of
> > > bind
> > > error)
> > > are you opening a new connection for adding each entry? it appears so
> to
> > me
> > > after seeing the method public static void
> > addEntryIntoSonarTestLdap(Entry
> > > labLdapEntry)
> > >
> > > > Thanks & Regards,
> > > > Jitesh Dundas
> > > >
> > > > Phone - 732-357-6292
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <
> kayyagari@apache.org
> > > > >wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jbdundas@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I am a new user who is implementing Apache LDAP on one of our
> > > > company's
> > > > > > servers. The records in the LDAP ( entries) come from a text
> file (
> > > > LDAP
> > > > > > dump) having 600000 records in it.
> > > > > >
> > > > > > I have written a utility to insert it buyt it keeps failing with
> > > these
> > > > > > errors:-
> > > > > >
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
> Bind
> > > > > failed
> > > > > > : timeout occurred
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
> The
> > > > > response
> > > > > > queue has been emptied, no response was found.
> > > > > > org.apache.directory.api.ldap.model.exception.LdapException:
> > TimeOut
> > > > > > occurred
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > > > at Main.getConnection(Main.java:38)
> > > > > > at Main.main(Main.java:130)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > > > >
> > > > > >
> > > > > > Here is the code to add it:-
> > > > > >
> > > > > > public static void addEntryIntoSonarTestLdap(Entry
> labLdapEntry)
> > > > throws
> > > > > > LdapException, IOException {
> > > > > > //Sonar Ldap Connection Details
> > > > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> > > "admin",
> > > > > > "secret");
> > > > > >
> > > > > replace "admin" with "uid=admin,ou=system" and try
> > > > >
> > > > > >
> > > > > > List<WebphonePerson> webphonePersons =
> > > > > WebphoneFileReader.readFile();
> > > > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > > > LdapConnection connection = client.getConnection();
> > > > > > connection.setTimeOut(1000000);
> > > > > > try {
> > > > > >
> > > > > >
> > > > > > //create the new entries to add.
> > > > > > Entry sonarLdapEntry = new
> > > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> > > entry
> > > > > goes
> > > > > > here.
> > > > > > sonarLdapEntry.add("objectClass","top");
> > > > > >
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > > > sonarLdapEntry.add("objectClass","person");
> > > > > >
> > > > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > > > sonarLdapEntry.add("cn",
> > > > > webphonePersons.get(i).getUID());
> > > > > >
> > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + "
> > " +
> > > > > > webphonePersons.get(i).getLastName());
> > > > > >
> > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > > > >
> > > > > >
> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > > > connection.add(sonarLdapEntry);
> > > > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > > > } catch (Exception e) {
> > > > > > // TODO Auto-generated catch block
> > > > > > //e.printStackTrace();
> > > > > > System.out.println("UID FAILED => "+
> > > > > > webphonePersons.get(i).getAttUID());
> > > > > > }
> > > > > > //closing connections after every 1000 records
> > > added
> > > > > and
> > > > > > then opened again.
> > > > > > //to prevent memory issues.
> > > > > > if (i%1000 == 0){
> > > > > > client.closeConnection(connection);
> > > > > > connection = client.getConnection();
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > Any ideas on how to resolve it? We have the Java memory options
> set
> > > to
> > > > > > -Xms1024m -Xmx1024m
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Jitesh Dundas
> > > > > >
> > > > > > Phone - 732-357-6292
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kiran Ayyagari
> > > > > http://keydap.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
Hi Kiran,
Thanks again for the reply. Appreciate it.
We are using the latest ApacheDS 2.0.0 version. I modified the method to
look like this:-
LdapConnection connection = client.getConnection();
connection.setTimeOut(1000000000);
Date date = new Date();
String dateStr = "Started="+date.getDate() + "-" + date.getMonth() +
"-" + date.getYear() ;
System.out.println("START@"+ dateStr );
List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
System.out.println("Found Users" + webphonePersons.size());
for(int i=18000; i<webphonePersons.size() ; i++){
try {
//create the new entries to add.
Entry sonarLdapEntry = new DefaultEntry("cn="+
webphonePersons.get(i).getAttUID() +",ou=people,O=ATT,C=US");//dn of the
entry goes here.
sonarLdapEntry.add("objectClass","top");
sonarLdapEntry.add("objectClass","inetOrgPerson");
sonarLdapEntry.add("objectClass","person");
sonarLdapEntry.add("objectClass","organizationalPerson");
sonarLdapEntry.add("cn",webphonePersons.get(i).getUID());
sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
webphonePersons.get(i).getLastName());
sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
connection.add(sonarLdapEntry);
Thread.sleep(200);
} catch (Exception e) {
//e.printStackTrace();
System.out.println("ATTUID FAILED => "+
webphonePersons.get(i).getAttUID());
}
if (i%1000 == 0 || webphonePersons.size() == i){
//client.closeConnection(connection);
System.out.println("Completed => "+ i);
Thread.sleep(4000);
//connection = client.getConnection();
}//end if condition
}//end for loop
Can you please tell me what is wrong here? I would love to make it better.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 11:26 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > HI Kiran,
> >
> > Thank you for your reply. You are correct in the sense that we close the
> > connection after every 1000 entries that are completed in the for loop
> and
> > then open the connection again.
> >
> > *if (i%1000 == 0 || webphonePersons.size() == i){*
> > client.closeConnection(connection);
> > System.out.println("Completed => "+ i);
> > Thread.sleep(4000);
> > connection = client.getConnection();
> > }//end if condition
> >
> > We are also putting Thread.sleep() to wait and give some time for the
> > connection to start/stop. Also, we have added the same sleep condition of
> > 400 milliseconds after every entry that is added( to prevent overlaps).
> >
> > However, based on your response, I am making the change to remove this
> > logic and keep one connection only throughout the loop. Hopefully it will
> > work.
> >
> > not that your method is wrong, but one connection is enough here
> otoh, which version of ApacheDS you are using?
>
> >
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
> >
> >
> >
> >
> > On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com>
> > wrote:
> > >
> > > > Hi Kiran,
> > > >
> > > > Thank you for your reply. My apologies for not being able to send the
> > > full
> > > > java file( As you can see that I am already doing that):-
> > > >
> > > > public Main(String host, int port, String userName, String
> password) {
> > > > this.host = host;
> > > > this.port = port;
> > > > this.userName = userName;
> > > > this.password = password;
> > > > }
> > > >
> > > > public LdapConnection getConnection() throws LdapException,
> > > IOException {
> > > > LdapConnection connection = new LdapNetworkConnection(host,
> port);
> > > > connection.bind("uid=" + userName + ",ou=system", password);
> > > > return connection;
> > > > }
> > > >
> > > > Also, I see from the updated logs now that it traverses 3000 records
> > and
> > > > then it starts giving these error messages. Any ideas?
> > > >
> > > > Maybe it is the way in which I am making the entries.
> > > >
> > > > if entry creation is wrong you will get a different error (instead of
> > > bind
> > > error)
> > > are you opening a new connection for adding each entry? it appears so
> to
> > me
> > > after seeing the method public static void
> > addEntryIntoSonarTestLdap(Entry
> > > labLdapEntry)
> > >
> > > > Thanks & Regards,
> > > > Jitesh Dundas
> > > >
> > > > Phone - 732-357-6292
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <
> kayyagari@apache.org
> > > > >wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jbdundas@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I am a new user who is implementing Apache LDAP on one of our
> > > > company's
> > > > > > servers. The records in the LDAP ( entries) come from a text
> file (
> > > > LDAP
> > > > > > dump) having 600000 records in it.
> > > > > >
> > > > > > I have written a utility to insert it buyt it keeps failing with
> > > these
> > > > > > errors:-
> > > > > >
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
> Bind
> > > > > failed
> > > > > > : timeout occurred
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
> The
> > > > > response
> > > > > > queue has been emptied, no response was found.
> > > > > > org.apache.directory.api.ldap.model.exception.LdapException:
> > TimeOut
> > > > > > occurred
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > > > at Main.getConnection(Main.java:38)
> > > > > > at Main.main(Main.java:130)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > > > >
> > > > > >
> > > > > > Here is the code to add it:-
> > > > > >
> > > > > > public static void addEntryIntoSonarTestLdap(Entry
> labLdapEntry)
> > > > throws
> > > > > > LdapException, IOException {
> > > > > > //Sonar Ldap Connection Details
> > > > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> > > "admin",
> > > > > > "secret");
> > > > > >
> > > > > replace "admin" with "uid=admin,ou=system" and try
> > > > >
> > > > > >
> > > > > > List<WebphonePerson> webphonePersons =
> > > > > WebphoneFileReader.readFile();
> > > > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > > > LdapConnection connection = client.getConnection();
> > > > > > connection.setTimeOut(1000000);
> > > > > > try {
> > > > > >
> > > > > >
> > > > > > //create the new entries to add.
> > > > > > Entry sonarLdapEntry = new
> > > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> > > entry
> > > > > goes
> > > > > > here.
> > > > > > sonarLdapEntry.add("objectClass","top");
> > > > > >
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > > > sonarLdapEntry.add("objectClass","person");
> > > > > >
> > > > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > > > sonarLdapEntry.add("cn",
> > > > > webphonePersons.get(i).getUID());
> > > > > >
> > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + "
> > " +
> > > > > > webphonePersons.get(i).getLastName());
> > > > > >
> > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > > > >
> > > > > >
> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > > > connection.add(sonarLdapEntry);
> > > > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > > > } catch (Exception e) {
> > > > > > // TODO Auto-generated catch block
> > > > > > //e.printStackTrace();
> > > > > > System.out.println("UID FAILED => "+
> > > > > > webphonePersons.get(i).getAttUID());
> > > > > > }
> > > > > > //closing connections after every 1000 records
> > > added
> > > > > and
> > > > > > then opened again.
> > > > > > //to prevent memory issues.
> > > > > > if (i%1000 == 0){
> > > > > > client.closeConnection(connection);
> > > > > > connection = client.getConnection();
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > Any ideas on how to resolve it? We have the Java memory options
> set
> > > to
> > > > > > -Xms1024m -Xmx1024m
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Jitesh Dundas
> > > > > >
> > > > > > Phone - 732-357-6292
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kiran Ayyagari
> > > > > http://keydap.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jb...@gmail.com> wrote:
> HI Kiran,
>
> Thank you for your reply. You are correct in the sense that we close the
> connection after every 1000 entries that are completed in the for loop and
> then open the connection again.
>
> *if (i%1000 == 0 || webphonePersons.size() == i){*
> client.closeConnection(connection);
> System.out.println("Completed => "+ i);
> Thread.sleep(4000);
> connection = client.getConnection();
> }//end if condition
>
> We are also putting Thread.sleep() to wait and give some time for the
> connection to start/stop. Also, we have added the same sleep condition of
> 400 milliseconds after every entry that is added( to prevent overlaps).
>
> However, based on your response, I am making the change to remove this
> logic and keep one connection only throughout the loop. Hopefully it will
> work.
>
> not that your method is wrong, but one connection is enough here
otoh, which version of ApacheDS you are using?
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com>
> wrote:
> >
> > > Hi Kiran,
> > >
> > > Thank you for your reply. My apologies for not being able to send the
> > full
> > > java file( As you can see that I am already doing that):-
> > >
> > > public Main(String host, int port, String userName, String password) {
> > > this.host = host;
> > > this.port = port;
> > > this.userName = userName;
> > > this.password = password;
> > > }
> > >
> > > public LdapConnection getConnection() throws LdapException,
> > IOException {
> > > LdapConnection connection = new LdapNetworkConnection(host, port);
> > > connection.bind("uid=" + userName + ",ou=system", password);
> > > return connection;
> > > }
> > >
> > > Also, I see from the updated logs now that it traverses 3000 records
> and
> > > then it starts giving these error messages. Any ideas?
> > >
> > > Maybe it is the way in which I am making the entries.
> > >
> > > if entry creation is wrong you will get a different error (instead of
> > bind
> > error)
> > are you opening a new connection for adding each entry? it appears so to
> me
> > after seeing the method public static void
> addEntryIntoSonarTestLdap(Entry
> > labLdapEntry)
> >
> > > Thanks & Regards,
> > > Jitesh Dundas
> > >
> > > Phone - 732-357-6292
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> > > >wrote:
> > >
> > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am a new user who is implementing Apache LDAP on one of our
> > > company's
> > > > > servers. The records in the LDAP ( entries) come from a text file (
> > > LDAP
> > > > > dump) having 600000 records in it.
> > > > >
> > > > > I have written a utility to insert it buyt it keeps failing with
> > these
> > > > > errors:-
> > > > >
> > > > > [10:16:25] ERROR
> > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > > > failed
> > > > > : timeout occurred
> > > > > [10:16:25] ERROR
> > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > > > response
> > > > > queue has been emptied, no response was found.
> > > > > org.apache.directory.api.ldap.model.exception.LdapException:
> TimeOut
> > > > > occurred
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > > at Main.getConnection(Main.java:38)
> > > > > at Main.main(Main.java:130)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > > >
> > > > >
> > > > > Here is the code to add it:-
> > > > >
> > > > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> > > throws
> > > > > LdapException, IOException {
> > > > > //Sonar Ldap Connection Details
> > > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> > "admin",
> > > > > "secret");
> > > > >
> > > > replace "admin" with "uid=admin,ou=system" and try
> > > >
> > > > >
> > > > > List<WebphonePerson> webphonePersons =
> > > > WebphoneFileReader.readFile();
> > > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > > LdapConnection connection = client.getConnection();
> > > > > connection.setTimeOut(1000000);
> > > > > try {
> > > > >
> > > > >
> > > > > //create the new entries to add.
> > > > > Entry sonarLdapEntry = new
> > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> > entry
> > > > goes
> > > > > here.
> > > > > sonarLdapEntry.add("objectClass","top");
> > > > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > > sonarLdapEntry.add("objectClass","person");
> > > > >
> > > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > > sonarLdapEntry.add("cn",
> > > > webphonePersons.get(i).getUID());
> > > > >
> > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + "
> " +
> > > > > webphonePersons.get(i).getLastName());
> > > > >
> > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > > >
> > > > >
> > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > > connection.add(sonarLdapEntry);
> > > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > > } catch (Exception e) {
> > > > > // TODO Auto-generated catch block
> > > > > //e.printStackTrace();
> > > > > System.out.println("UID FAILED => "+
> > > > > webphonePersons.get(i).getAttUID());
> > > > > }
> > > > > //closing connections after every 1000 records
> > added
> > > > and
> > > > > then opened again.
> > > > > //to prevent memory issues.
> > > > > if (i%1000 == 0){
> > > > > client.closeConnection(connection);
> > > > > connection = client.getConnection();
> > > > > }
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > Any ideas on how to resolve it? We have the Java memory options set
> > to
> > > > > -Xms1024m -Xmx1024m
> > > > >
> > > > > Thanks & Regards,
> > > > > Jitesh Dundas
> > > > >
> > > > > Phone - 732-357-6292
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kiran Ayyagari
> > > > http://keydap.com
> > > >
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jb...@gmail.com> wrote:
> HI Kiran,
>
> Thank you for your reply. You are correct in the sense that we close the
> connection after every 1000 entries that are completed in the for loop and
> then open the connection again.
>
> *if (i%1000 == 0 || webphonePersons.size() == i){*
> client.closeConnection(connection);
> System.out.println("Completed => "+ i);
> Thread.sleep(4000);
> connection = client.getConnection();
> }//end if condition
>
> We are also putting Thread.sleep() to wait and give some time for the
> connection to start/stop. Also, we have added the same sleep condition of
> 400 milliseconds after every entry that is added( to prevent overlaps).
>
> However, based on your response, I am making the change to remove this
> logic and keep one connection only throughout the loop. Hopefully it will
> work.
>
> not that your method is wrong, but one connection is enough here
otoh, which version of ApacheDS you are using?
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com>
> wrote:
> >
> > > Hi Kiran,
> > >
> > > Thank you for your reply. My apologies for not being able to send the
> > full
> > > java file( As you can see that I am already doing that):-
> > >
> > > public Main(String host, int port, String userName, String password) {
> > > this.host = host;
> > > this.port = port;
> > > this.userName = userName;
> > > this.password = password;
> > > }
> > >
> > > public LdapConnection getConnection() throws LdapException,
> > IOException {
> > > LdapConnection connection = new LdapNetworkConnection(host, port);
> > > connection.bind("uid=" + userName + ",ou=system", password);
> > > return connection;
> > > }
> > >
> > > Also, I see from the updated logs now that it traverses 3000 records
> and
> > > then it starts giving these error messages. Any ideas?
> > >
> > > Maybe it is the way in which I am making the entries.
> > >
> > > if entry creation is wrong you will get a different error (instead of
> > bind
> > error)
> > are you opening a new connection for adding each entry? it appears so to
> me
> > after seeing the method public static void
> addEntryIntoSonarTestLdap(Entry
> > labLdapEntry)
> >
> > > Thanks & Regards,
> > > Jitesh Dundas
> > >
> > > Phone - 732-357-6292
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> > > >wrote:
> > >
> > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am a new user who is implementing Apache LDAP on one of our
> > > company's
> > > > > servers. The records in the LDAP ( entries) come from a text file (
> > > LDAP
> > > > > dump) having 600000 records in it.
> > > > >
> > > > > I have written a utility to insert it buyt it keeps failing with
> > these
> > > > > errors:-
> > > > >
> > > > > [10:16:25] ERROR
> > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > > > failed
> > > > > : timeout occurred
> > > > > [10:16:25] ERROR
> > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > > > response
> > > > > queue has been emptied, no response was found.
> > > > > org.apache.directory.api.ldap.model.exception.LdapException:
> TimeOut
> > > > > occurred
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > > at Main.getConnection(Main.java:38)
> > > > > at Main.main(Main.java:130)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > > >
> > > > >
> > > > > Here is the code to add it:-
> > > > >
> > > > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> > > throws
> > > > > LdapException, IOException {
> > > > > //Sonar Ldap Connection Details
> > > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> > "admin",
> > > > > "secret");
> > > > >
> > > > replace "admin" with "uid=admin,ou=system" and try
> > > >
> > > > >
> > > > > List<WebphonePerson> webphonePersons =
> > > > WebphoneFileReader.readFile();
> > > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > > LdapConnection connection = client.getConnection();
> > > > > connection.setTimeOut(1000000);
> > > > > try {
> > > > >
> > > > >
> > > > > //create the new entries to add.
> > > > > Entry sonarLdapEntry = new
> > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> > entry
> > > > goes
> > > > > here.
> > > > > sonarLdapEntry.add("objectClass","top");
> > > > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > > sonarLdapEntry.add("objectClass","person");
> > > > >
> > > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > > sonarLdapEntry.add("cn",
> > > > webphonePersons.get(i).getUID());
> > > > >
> > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + "
> " +
> > > > > webphonePersons.get(i).getLastName());
> > > > >
> > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > > >
> > > > >
> > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > > connection.add(sonarLdapEntry);
> > > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > > } catch (Exception e) {
> > > > > // TODO Auto-generated catch block
> > > > > //e.printStackTrace();
> > > > > System.out.println("UID FAILED => "+
> > > > > webphonePersons.get(i).getAttUID());
> > > > > }
> > > > > //closing connections after every 1000 records
> > added
> > > > and
> > > > > then opened again.
> > > > > //to prevent memory issues.
> > > > > if (i%1000 == 0){
> > > > > client.closeConnection(connection);
> > > > > connection = client.getConnection();
> > > > > }
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > Any ideas on how to resolve it? We have the Java memory options set
> > to
> > > > > -Xms1024m -Xmx1024m
> > > > >
> > > > > Thanks & Regards,
> > > > > Jitesh Dundas
> > > > >
> > > > > Phone - 732-357-6292
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kiran Ayyagari
> > > > http://keydap.com
> > > >
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
HI Kiran,
Thank you for your reply. You are correct in the sense that we close the
connection after every 1000 entries that are completed in the for loop and
then open the connection again.
*if (i%1000 == 0 || webphonePersons.size() == i){*
client.closeConnection(connection);
System.out.println("Completed => "+ i);
Thread.sleep(4000);
connection = client.getConnection();
}//end if condition
We are also putting Thread.sleep() to wait and give some time for the
connection to start/stop. Also, we have added the same sleep condition of
400 milliseconds after every entry that is added( to prevent overlaps).
However, based on your response, I am making the change to remove this
logic and keep one connection only throughout the loop. Hopefully it will
work.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > Hi Kiran,
> >
> > Thank you for your reply. My apologies for not being able to send the
> full
> > java file( As you can see that I am already doing that):-
> >
> > public Main(String host, int port, String userName, String password) {
> > this.host = host;
> > this.port = port;
> > this.userName = userName;
> > this.password = password;
> > }
> >
> > public LdapConnection getConnection() throws LdapException,
> IOException {
> > LdapConnection connection = new LdapNetworkConnection(host, port);
> > connection.bind("uid=" + userName + ",ou=system", password);
> > return connection;
> > }
> >
> > Also, I see from the updated logs now that it traverses 3000 records and
> > then it starts giving these error messages. Any ideas?
> >
> > Maybe it is the way in which I am making the entries.
> >
> > if entry creation is wrong you will get a different error (instead of
> bind
> error)
> are you opening a new connection for adding each entry? it appears so to me
> after seeing the method public static void addEntryIntoSonarTestLdap(Entry
> labLdapEntry)
>
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
> >
> >
> >
> >
> > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am a new user who is implementing Apache LDAP on one of our
> > company's
> > > > servers. The records in the LDAP ( entries) come from a text file (
> > LDAP
> > > > dump) having 600000 records in it.
> > > >
> > > > I have written a utility to insert it buyt it keeps failing with
> these
> > > > errors:-
> > > >
> > > > [10:16:25] ERROR
> > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > > failed
> > > > : timeout occurred
> > > > [10:16:25] ERROR
> > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > > response
> > > > queue has been emptied, no response was found.
> > > > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > > > occurred
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > at Main.getConnection(Main.java:38)
> > > > at Main.main(Main.java:130)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > >
> > > >
> > > > Here is the code to add it:-
> > > >
> > > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> > throws
> > > > LdapException, IOException {
> > > > //Sonar Ldap Connection Details
> > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> "admin",
> > > > "secret");
> > > >
> > > replace "admin" with "uid=admin,ou=system" and try
> > >
> > > >
> > > > List<WebphonePerson> webphonePersons =
> > > WebphoneFileReader.readFile();
> > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > LdapConnection connection = client.getConnection();
> > > > connection.setTimeOut(1000000);
> > > > try {
> > > >
> > > >
> > > > //create the new entries to add.
> > > > Entry sonarLdapEntry = new
> > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> entry
> > > goes
> > > > here.
> > > > sonarLdapEntry.add("objectClass","top");
> > > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > sonarLdapEntry.add("objectClass","person");
> > > >
> > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > sonarLdapEntry.add("cn",
> > > webphonePersons.get(i).getUID());
> > > >
> > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > > > webphonePersons.get(i).getLastName());
> > > >
> > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > >
> > > >
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > connection.add(sonarLdapEntry);
> > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > } catch (Exception e) {
> > > > // TODO Auto-generated catch block
> > > > //e.printStackTrace();
> > > > System.out.println("UID FAILED => "+
> > > > webphonePersons.get(i).getAttUID());
> > > > }
> > > > //closing connections after every 1000 records
> added
> > > and
> > > > then opened again.
> > > > //to prevent memory issues.
> > > > if (i%1000 == 0){
> > > > client.closeConnection(connection);
> > > > connection = client.getConnection();
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > Any ideas on how to resolve it? We have the Java memory options set
> to
> > > > -Xms1024m -Xmx1024m
> > > >
> > > > Thanks & Regards,
> > > > Jitesh Dundas
> > > >
> > > > Phone - 732-357-6292
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
HI Kiran,
Thank you for your reply. You are correct in the sense that we close the
connection after every 1000 entries that are completed in the for loop and
then open the connection again.
*if (i%1000 == 0 || webphonePersons.size() == i){*
client.closeConnection(connection);
System.out.println("Completed => "+ i);
Thread.sleep(4000);
connection = client.getConnection();
}//end if condition
We are also putting Thread.sleep() to wait and give some time for the
connection to start/stop. Also, we have added the same sleep condition of
400 milliseconds after every entry that is added( to prevent overlaps).
However, based on your response, I am making the change to remove this
logic and keep one connection only throughout the loop. Hopefully it will
work.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > Hi Kiran,
> >
> > Thank you for your reply. My apologies for not being able to send the
> full
> > java file( As you can see that I am already doing that):-
> >
> > public Main(String host, int port, String userName, String password) {
> > this.host = host;
> > this.port = port;
> > this.userName = userName;
> > this.password = password;
> > }
> >
> > public LdapConnection getConnection() throws LdapException,
> IOException {
> > LdapConnection connection = new LdapNetworkConnection(host, port);
> > connection.bind("uid=" + userName + ",ou=system", password);
> > return connection;
> > }
> >
> > Also, I see from the updated logs now that it traverses 3000 records and
> > then it starts giving these error messages. Any ideas?
> >
> > Maybe it is the way in which I am making the entries.
> >
> > if entry creation is wrong you will get a different error (instead of
> bind
> error)
> are you opening a new connection for adding each entry? it appears so to me
> after seeing the method public static void addEntryIntoSonarTestLdap(Entry
> labLdapEntry)
>
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
> >
> >
> >
> >
> > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am a new user who is implementing Apache LDAP on one of our
> > company's
> > > > servers. The records in the LDAP ( entries) come from a text file (
> > LDAP
> > > > dump) having 600000 records in it.
> > > >
> > > > I have written a utility to insert it buyt it keeps failing with
> these
> > > > errors:-
> > > >
> > > > [10:16:25] ERROR
> > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > > failed
> > > > : timeout occurred
> > > > [10:16:25] ERROR
> > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > > response
> > > > queue has been emptied, no response was found.
> > > > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > > > occurred
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > at Main.getConnection(Main.java:38)
> > > > at Main.main(Main.java:130)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> > > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > >
> > > >
> > > > Here is the code to add it:-
> > > >
> > > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> > throws
> > > > LdapException, IOException {
> > > > //Sonar Ldap Connection Details
> > > > Main client = new Main("altv0018.aldc.att.com", 10389,
> "admin",
> > > > "secret");
> > > >
> > > replace "admin" with "uid=admin,ou=system" and try
> > >
> > > >
> > > > List<WebphonePerson> webphonePersons =
> > > WebphoneFileReader.readFile();
> > > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > > LdapConnection connection = client.getConnection();
> > > > connection.setTimeOut(1000000);
> > > > try {
> > > >
> > > >
> > > > //create the new entries to add.
> > > > Entry sonarLdapEntry = new
> > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the
> entry
> > > goes
> > > > here.
> > > > sonarLdapEntry.add("objectClass","top");
> > > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > sonarLdapEntry.add("objectClass","person");
> > > >
> > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > sonarLdapEntry.add("cn",
> > > webphonePersons.get(i).getUID());
> > > >
> > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > > > webphonePersons.get(i).getLastName());
> > > >
> > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > >
> > > >
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > connection.add(sonarLdapEntry);
> > > > LOGGER.info("Sonar Test LDAP@Added User
> > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > } catch (Exception e) {
> > > > // TODO Auto-generated catch block
> > > > //e.printStackTrace();
> > > > System.out.println("UID FAILED => "+
> > > > webphonePersons.get(i).getAttUID());
> > > > }
> > > > //closing connections after every 1000 records
> added
> > > and
> > > > then opened again.
> > > > //to prevent memory issues.
> > > > if (i%1000 == 0){
> > > > client.closeConnection(connection);
> > > > connection = client.getConnection();
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > Any ideas on how to resolve it? We have the Java memory options set
> to
> > > > -Xms1024m -Xmx1024m
> > > >
> > > > Thanks & Regards,
> > > > Jitesh Dundas
> > > >
> > > > Phone - 732-357-6292
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com> wrote:
> Hi Kiran,
>
> Thank you for your reply. My apologies for not being able to send the full
> java file( As you can see that I am already doing that):-
>
> public Main(String host, int port, String userName, String password) {
> this.host = host;
> this.port = port;
> this.userName = userName;
> this.password = password;
> }
>
> public LdapConnection getConnection() throws LdapException, IOException {
> LdapConnection connection = new LdapNetworkConnection(host, port);
> connection.bind("uid=" + userName + ",ou=system", password);
> return connection;
> }
>
> Also, I see from the updated logs now that it traverses 3000 records and
> then it starts giving these error messages. Any ideas?
>
> Maybe it is the way in which I am making the entries.
>
> if entry creation is wrong you will get a different error (instead of bind
error)
are you opening a new connection for adding each entry? it appears so to me
after seeing the method public static void addEntryIntoSonarTestLdap(Entry
labLdapEntry)
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I am a new user who is implementing Apache LDAP on one of our
> company's
> > > servers. The records in the LDAP ( entries) come from a text file (
> LDAP
> > > dump) having 600000 records in it.
> > >
> > > I have written a utility to insert it buyt it keeps failing with these
> > > errors:-
> > >
> > > [10:16:25] ERROR
> > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > failed
> > > : timeout occurred
> > > [10:16:25] ERROR
> > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > response
> > > queue has been emptied, no response was found.
> > > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > > occurred
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > at Main.getConnection(Main.java:38)
> > > at Main.main(Main.java:130)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > >
> > >
> > > Here is the code to add it:-
> > >
> > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> throws
> > > LdapException, IOException {
> > > //Sonar Ldap Connection Details
> > > Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> > > "secret");
> > >
> > replace "admin" with "uid=admin,ou=system" and try
> >
> > >
> > > List<WebphonePerson> webphonePersons =
> > WebphoneFileReader.readFile();
> > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > LdapConnection connection = client.getConnection();
> > > connection.setTimeOut(1000000);
> > > try {
> > >
> > >
> > > //create the new entries to add.
> > > Entry sonarLdapEntry = new
> > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry
> > goes
> > > here.
> > > sonarLdapEntry.add("objectClass","top");
> > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > sonarLdapEntry.add("objectClass","person");
> > >
> sonarLdapEntry.add("objectClass","organizationalPerson");
> > > sonarLdapEntry.add("cn",
> > webphonePersons.get(i).getUID());
> > >
> > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > > webphonePersons.get(i).getLastName());
> > >
> > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > >
> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > connection.add(sonarLdapEntry);
> > > LOGGER.info("Sonar Test LDAP@Added User
> > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > } catch (Exception e) {
> > > // TODO Auto-generated catch block
> > > //e.printStackTrace();
> > > System.out.println("UID FAILED => "+
> > > webphonePersons.get(i).getAttUID());
> > > }
> > > //closing connections after every 1000 records added
> > and
> > > then opened again.
> > > //to prevent memory issues.
> > > if (i%1000 == 0){
> > > client.closeConnection(connection);
> > > connection = client.getConnection();
> > > }
> > > }
> > >
> > > }
> > >
> > > Any ideas on how to resolve it? We have the Java memory options set to
> > > -Xms1024m -Xmx1024m
> > >
> > > Thanks & Regards,
> > > Jitesh Dundas
> > >
> > > Phone - 732-357-6292
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jb...@gmail.com> wrote:
> Hi Kiran,
>
> Thank you for your reply. My apologies for not being able to send the full
> java file( As you can see that I am already doing that):-
>
> public Main(String host, int port, String userName, String password) {
> this.host = host;
> this.port = port;
> this.userName = userName;
> this.password = password;
> }
>
> public LdapConnection getConnection() throws LdapException, IOException {
> LdapConnection connection = new LdapNetworkConnection(host, port);
> connection.bind("uid=" + userName + ",ou=system", password);
> return connection;
> }
>
> Also, I see from the updated logs now that it traverses 3000 records and
> then it starts giving these error messages. Any ideas?
>
> Maybe it is the way in which I am making the entries.
>
> if entry creation is wrong you will get a different error (instead of bind
error)
are you opening a new connection for adding each entry? it appears so to me
after seeing the method public static void addEntryIntoSonarTestLdap(Entry
labLdapEntry)
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I am a new user who is implementing Apache LDAP on one of our
> company's
> > > servers. The records in the LDAP ( entries) come from a text file (
> LDAP
> > > dump) having 600000 records in it.
> > >
> > > I have written a utility to insert it buyt it keeps failing with these
> > > errors:-
> > >
> > > [10:16:25] ERROR
> > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> > failed
> > > : timeout occurred
> > > [10:16:25] ERROR
> > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> > response
> > > queue has been emptied, no response was found.
> > > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > > occurred
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > at
> > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > at Main.getConnection(Main.java:38)
> > > at Main.main(Main.java:130)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> > > at java.lang.reflect.Method.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > >
> > >
> > > Here is the code to add it:-
> > >
> > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry)
> throws
> > > LdapException, IOException {
> > > //Sonar Ldap Connection Details
> > > Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> > > "secret");
> > >
> > replace "admin" with "uid=admin,ou=system" and try
> >
> > >
> > > List<WebphonePerson> webphonePersons =
> > WebphoneFileReader.readFile();
> > > for(int i=6000; i<webphonePersons.size() ; i++){
> > > LdapConnection connection = client.getConnection();
> > > connection.setTimeOut(1000000);
> > > try {
> > >
> > >
> > > //create the new entries to add.
> > > Entry sonarLdapEntry = new
> > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry
> > goes
> > > here.
> > > sonarLdapEntry.add("objectClass","top");
> > > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > sonarLdapEntry.add("objectClass","person");
> > >
> sonarLdapEntry.add("objectClass","organizationalPerson");
> > > sonarLdapEntry.add("cn",
> > webphonePersons.get(i).getUID());
> > >
> > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > > webphonePersons.get(i).getLastName());
> > >
> > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > >
> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > connection.add(sonarLdapEntry);
> > > LOGGER.info("Sonar Test LDAP@Added User
> > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > } catch (Exception e) {
> > > // TODO Auto-generated catch block
> > > //e.printStackTrace();
> > > System.out.println("UID FAILED => "+
> > > webphonePersons.get(i).getAttUID());
> > > }
> > > //closing connections after every 1000 records added
> > and
> > > then opened again.
> > > //to prevent memory issues.
> > > if (i%1000 == 0){
> > > client.closeConnection(connection);
> > > connection = client.getConnection();
> > > }
> > > }
> > >
> > > }
> > >
> > > Any ideas on how to resolve it? We have the Java memory options set to
> > > -Xms1024m -Xmx1024m
> > >
> > > Thanks & Regards,
> > > Jitesh Dundas
> > >
> > > Phone - 732-357-6292
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
Hi Kiran,
Thank you for your reply. My apologies for not being able to send the full
java file( As you can see that I am already doing that):-
public Main(String host, int port, String userName, String password) {
this.host = host;
this.port = port;
this.userName = userName;
this.password = password;
}
public LdapConnection getConnection() throws LdapException, IOException {
LdapConnection connection = new LdapNetworkConnection(host, port);
connection.bind("uid=" + userName + ",ou=system", password);
return connection;
}
Also, I see from the updated logs now that it traverses 3000 records and
then it starts giving these error messages. Any ideas?
Maybe it is the way in which I am making the entries.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am a new user who is implementing Apache LDAP on one of our company's
> > servers. The records in the LDAP ( entries) come from a text file ( LDAP
> > dump) having 600000 records in it.
> >
> > I have written a utility to insert it buyt it keeps failing with these
> > errors:-
> >
> > [10:16:25] ERROR
> > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> failed
> > : timeout occurred
> > [10:16:25] ERROR
> > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> response
> > queue has been emptied, no response was found.
> > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > occurred
> > at
> >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > at
> >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > at
> >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > at Main.getConnection(Main.java:38)
> > at Main.main(Main.java:130)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> >
> >
> > Here is the code to add it:-
> >
> > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) throws
> > LdapException, IOException {
> > //Sonar Ldap Connection Details
> > Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> > "secret");
> >
> replace "admin" with "uid=admin,ou=system" and try
>
> >
> > List<WebphonePerson> webphonePersons =
> WebphoneFileReader.readFile();
> > for(int i=6000; i<webphonePersons.size() ; i++){
> > LdapConnection connection = client.getConnection();
> > connection.setTimeOut(1000000);
> > try {
> >
> >
> > //create the new entries to add.
> > Entry sonarLdapEntry = new
> > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry
> goes
> > here.
> > sonarLdapEntry.add("objectClass","top");
> > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > sonarLdapEntry.add("objectClass","person");
> > sonarLdapEntry.add("objectClass","organizationalPerson");
> > sonarLdapEntry.add("cn",
> webphonePersons.get(i).getUID());
> >
> > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > webphonePersons.get(i).getLastName());
> >
> sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> >
> > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > connection.add(sonarLdapEntry);
> > LOGGER.info("Sonar Test LDAP@Added User
> > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > } catch (Exception e) {
> > // TODO Auto-generated catch block
> > //e.printStackTrace();
> > System.out.println("UID FAILED => "+
> > webphonePersons.get(i).getAttUID());
> > }
> > //closing connections after every 1000 records added
> and
> > then opened again.
> > //to prevent memory issues.
> > if (i%1000 == 0){
> > client.closeConnection(connection);
> > connection = client.getConnection();
> > }
> > }
> >
> > }
> >
> > Any ideas on how to resolve it? We have the Java memory options set to
> > -Xms1024m -Xmx1024m
> >
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by jitesh dundas <jb...@gmail.com>.
Hi Kiran,
Thank you for your reply. My apologies for not being able to send the full
java file( As you can see that I am already doing that):-
public Main(String host, int port, String userName, String password) {
this.host = host;
this.port = port;
this.userName = userName;
this.password = password;
}
public LdapConnection getConnection() throws LdapException, IOException {
LdapConnection connection = new LdapNetworkConnection(host, port);
connection.bind("uid=" + userName + ",ou=system", password);
return connection;
}
Also, I see from the updated logs now that it traverses 3000 records and
then it starts giving these error messages. Any ideas?
Maybe it is the way in which I am making the entries.
Thanks & Regards,
Jitesh Dundas
Phone - 732-357-6292
On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am a new user who is implementing Apache LDAP on one of our company's
> > servers. The records in the LDAP ( entries) come from a text file ( LDAP
> > dump) having 600000 records in it.
> >
> > I have written a utility to insert it buyt it keeps failing with these
> > errors:-
> >
> > [10:16:25] ERROR
> > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind
> failed
> > : timeout occurred
> > [10:16:25] ERROR
> > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The
> response
> > queue has been emptied, no response was found.
> > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> > occurred
> > at
> >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > at
> >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > at
> >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > at Main.getConnection(Main.java:38)
> > at Main.main(Main.java:130)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> >
> >
> > Here is the code to add it:-
> >
> > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) throws
> > LdapException, IOException {
> > //Sonar Ldap Connection Details
> > Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> > "secret");
> >
> replace "admin" with "uid=admin,ou=system" and try
>
> >
> > List<WebphonePerson> webphonePersons =
> WebphoneFileReader.readFile();
> > for(int i=6000; i<webphonePersons.size() ; i++){
> > LdapConnection connection = client.getConnection();
> > connection.setTimeOut(1000000);
> > try {
> >
> >
> > //create the new entries to add.
> > Entry sonarLdapEntry = new
> > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry
> goes
> > here.
> > sonarLdapEntry.add("objectClass","top");
> > sonarLdapEntry.add("objectClass","inetOrgPerson");
> > sonarLdapEntry.add("objectClass","person");
> > sonarLdapEntry.add("objectClass","organizationalPerson");
> > sonarLdapEntry.add("cn",
> webphonePersons.get(i).getUID());
> >
> > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> > webphonePersons.get(i).getLastName());
> >
> sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> >
> > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > connection.add(sonarLdapEntry);
> > LOGGER.info("Sonar Test LDAP@Added User
> > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > } catch (Exception e) {
> > // TODO Auto-generated catch block
> > //e.printStackTrace();
> > System.out.println("UID FAILED => "+
> > webphonePersons.get(i).getAttUID());
> > }
> > //closing connections after every 1000 records added
> and
> > then opened again.
> > //to prevent memory issues.
> > if (i%1000 == 0){
> > client.closeConnection(connection);
> > connection = client.getConnection();
> > }
> > }
> >
> > }
> >
> > Any ideas on how to resolve it? We have the Java memory options set to
> > -Xms1024m -Xmx1024m
> >
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com> wrote:
> Hi All,
>
> I am a new user who is implementing Apache LDAP on one of our company's
> servers. The records in the LDAP ( entries) come from a text file ( LDAP
> dump) having 600000 records in it.
>
> I have written a utility to insert it buyt it keeps failing with these
> errors:-
>
> [10:16:25] ERROR
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind failed
> : timeout occurred
> [10:16:25] ERROR
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The response
> queue has been emptied, no response was found.
> org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> occurred
> at
>
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> at
>
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> at
>
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> at Main.getConnection(Main.java:38)
> at Main.main(Main.java:130)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
>
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
>
>
> Here is the code to add it:-
>
> public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) throws
> LdapException, IOException {
> //Sonar Ldap Connection Details
> Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> "secret");
>
replace "admin" with "uid=admin,ou=system" and try
>
> List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
> for(int i=6000; i<webphonePersons.size() ; i++){
> LdapConnection connection = client.getConnection();
> connection.setTimeOut(1000000);
> try {
>
>
> //create the new entries to add.
> Entry sonarLdapEntry = new
> DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry goes
> here.
> sonarLdapEntry.add("objectClass","top");
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> sonarLdapEntry.add("objectClass","person");
> sonarLdapEntry.add("objectClass","organizationalPerson");
> sonarLdapEntry.add("cn", webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> webphonePersons.get(i).getLastName());
> sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> connection.add(sonarLdapEntry);
> LOGGER.info("Sonar Test LDAP@Added User
> sonarLdapEntry=>"+sonarLdapEntry.toString());
> } catch (Exception e) {
> // TODO Auto-generated catch block
> //e.printStackTrace();
> System.out.println("UID FAILED => "+
> webphonePersons.get(i).getAttUID());
> }
> //closing connections after every 1000 records added and
> then opened again.
> //to prevent memory issues.
> if (i%1000 == 0){
> client.closeConnection(connection);
> connection = client.getConnection();
> }
> }
>
> }
>
> Any ideas on how to resolve it? We have the Java memory options set to
> -Xms1024m -Xmx1024m
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
--
Kiran Ayyagari
http://keydap.com
Re: Need Help with the LDAP API for large records
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jb...@gmail.com> wrote:
> Hi All,
>
> I am a new user who is implementing Apache LDAP on one of our company's
> servers. The records in the LDAP ( entries) come from a text file ( LDAP
> dump) having 600000 records in it.
>
> I have written a utility to insert it buyt it keeps failing with these
> errors:-
>
> [10:16:25] ERROR
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind failed
> : timeout occurred
> [10:16:25] ERROR
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The response
> queue has been emptied, no response was found.
> org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
> occurred
> at
>
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> at
>
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> at
>
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> at Main.getConnection(Main.java:38)
> at Main.main(Main.java:130)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
>
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
>
>
> Here is the code to add it:-
>
> public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) throws
> LdapException, IOException {
> //Sonar Ldap Connection Details
> Main client = new Main("altv0018.aldc.att.com", 10389, "admin",
> "secret");
>
replace "admin" with "uid=admin,ou=system" and try
>
> List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
> for(int i=6000; i<webphonePersons.size() ; i++){
> LdapConnection connection = client.getConnection();
> connection.setTimeOut(1000000);
> try {
>
>
> //create the new entries to add.
> Entry sonarLdapEntry = new
> DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the entry goes
> here.
> sonarLdapEntry.add("objectClass","top");
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> sonarLdapEntry.add("objectClass","person");
> sonarLdapEntry.add("objectClass","organizationalPerson");
> sonarLdapEntry.add("cn", webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> webphonePersons.get(i).getLastName());
> sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> connection.add(sonarLdapEntry);
> LOGGER.info("Sonar Test LDAP@Added User
> sonarLdapEntry=>"+sonarLdapEntry.toString());
> } catch (Exception e) {
> // TODO Auto-generated catch block
> //e.printStackTrace();
> System.out.println("UID FAILED => "+
> webphonePersons.get(i).getAttUID());
> }
> //closing connections after every 1000 records added and
> then opened again.
> //to prevent memory issues.
> if (i%1000 == 0){
> client.closeConnection(connection);
> connection = client.getConnection();
> }
> }
>
> }
>
> Any ideas on how to resolve it? We have the Java memory options set to
> -Xms1024m -Xmx1024m
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
--
Kiran Ayyagari
http://keydap.com