You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Sebu Koleth (JIRA)" <ji...@apache.org> on 2010/04/26 22:30:32 UTC
[jira] Created: (DIRAPI-13) Recursively drilling into the directory
structure causes java.lang.OutOfMemoryError
Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
-----------------------------------------------------------------------------------
Key: DIRAPI-13
URL: https://issues.apache.org/jira/browse/DIRAPI-13
Project: Directory client API
Issue Type: Bug
Affects Versions: 0.1.0
Environment: Windows XP SP3
Reporter: Sebu Koleth
private static void recursivelyDescend(LdapConnection connection, String dn) {
System.out.println("Searching for children of dn : " + dn);
try
{
Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
while (cursor.next())
{
SearchResponse response = cursor.get();
if(response instanceof SearchResultEntry) {
recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
} else {
System.out.println("Unusable response type " + response);
}
}
} catch (LdapException le) {
le.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRAPI-13) Recursively drilling into the
directory structure causes java.lang.OutOfMemoryError
Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861086#action_12861086 ]
Kiran Ayyagari commented on DIRAPI-13:
--------------------------------------
Recursively browsing a directory containing large number of entries is not a good idea cause the cursors will be left open (and the client keep receiving the
entries) which will definitely lead to OOM.
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRAPI-13) Recursively drilling into the
directory structure causes java.lang.OutOfMemoryError
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861258#action_12861258 ]
Emmanuel Lecharny commented on DIRAPI-13:
-----------------------------------------
The code should work. The API need to be checked and those errors (ie PDU has been fully decoded ...) have to be fixed.
We wil investigate what's going on.
Thanks for the report !
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRAPI-13) Recursively drilling into the directory
structure causes java.lang.OutOfMemoryError
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRAPI-13:
------------------------------------
Affects Version/s: 0.2.0
(was: 0.1.0)
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRAPI-13) Recursively drilling into the directory
structure causes java.lang.OutOfMemoryError
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny updated DIRAPI-13:
------------------------------------
Fix Version/s: 0.2.0
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
> Fix For: 0.2.0
>
> Attachments: ApacheLdapAPITest.java, UnboundidLdapAPITest.java
>
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DIRAPI-13) Recursively drilling into the directory
structure causes java.lang.OutOfMemoryError
Posted by "Sebu Koleth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebu Koleth updated DIRAPI-13:
------------------------------
Attachment: ApacheLdapAPITest.java
UnboundidLdapAPITest.java
Attaching the classes used for testing Apache LDAP Client API and Unboundid LDAP SDK.
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
> Attachments: ApacheLdapAPITest.java, UnboundidLdapAPITest.java
>
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRAPI-13) Recursively drilling into the
directory structure causes java.lang.OutOfMemoryError
Posted by "Sebu Koleth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861211#action_12861211 ]
Sebu Koleth commented on DIRAPI-13:
-----------------------------------
In contrast I have been using plain JNDI code with a few enhancements to do the same thing and it works without any problem. So does the Unboundid LDAP SDK. So would a re-think of the LDAP Client API be in order?
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRAPI-13) Recursively drilling into the
directory structure causes java.lang.OutOfMemoryError
Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRAPI-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861264#action_12861264 ]
Kiran Ayyagari commented on DIRAPI-13:
--------------------------------------
Note that you need to close the cursors otherwise they still keep receiving the entries from the server and stored in memory. Though you think the program
failed just after processing 52 entries, behind the scenes the earlier opened cursors are still receiving entries, note that your program has opened 52
cursors and all are left open!.
It would be good to know how you are performing the same task with other APIs, do you mind attaching the code as a test case?
> Recursively drilling into the directory structure causes java.lang.OutOfMemoryError
> -----------------------------------------------------------------------------------
>
> Key: DIRAPI-13
> URL: https://issues.apache.org/jira/browse/DIRAPI-13
> Project: Directory client API
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Windows XP SP3
> Reporter: Sebu Koleth
>
> private static void recursivelyDescend(LdapConnection connection, String dn) {
> System.out.println("Searching for children of dn : " + dn);
> try
> {
> Cursor<SearchResponse> cursor = connection.search(dn, "(objectclass=*)", SearchScope.ONELEVEL, "*");
> while (cursor.next())
> {
> SearchResponse response = cursor.get();
> if(response instanceof SearchResultEntry) {
> recursivelyDescend(connection, ((SearchResultEntry)response).getObjectName().getName());
> } else {
> System.out.println("Unusable response type " + response);
> }
> }
> } catch (LdapException le) {
> le.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> The above piece of code is exercised after obtaining an SSL-based LDAP connection. The target server has hundreds of thousands of records at different levels. Logging at WARN level shows a *lot* of messages :
> WARN NioProcessor-1 org.apache.directory.shared.asn1.ber.Asn1Decoder - The PDU has been fully decoded but there are still bytes in the buffer.
> The code chokes at processing the 52nd entry that is two levels deep from the base DN. At this level there are around 1000 sub-levels.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.