You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Martin Alderson <eq...@planetquake.com> on 2009/03/25 18:07:18 UTC

[ApacheDS] OutOfMemoryError

Hi all,

I'm just running some tests on ApacheDS 1.5.4 on Windows.

I have added 25000 simple users to ou=users,ou=system [1].  After 
running the following LDAP search command I get between 1789 and 5179 
results each time before an OutOfMemoryError is produced by the server.

I'm not really sure why this is happening - shouldn't each search result 
entry be discarded after it has been sent back to the client?

Thanks,

Martin


[1] The format of the users added is:
dn: uid=user1,ou=users,ou=system
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: top
cn: user1
displayname: test user
sn: test
uid: test1

Re: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <eq...@planetquake.com>.
> Ok, so we still have a serious issu we have to fix quickly...
> 
> Can you give us the information about the used JVM, memory used and such 
> so that we can reproduce the problem easilly ?
> 
> May be we should reopen the JIRA too.
> 
> Thanks !

Sun JVM on Windows:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)

I'm only running 1.5 so I can use JMP though - I have also run tests 
with 1.6 and had the same problem.

I'm starting the server with default settings - which I think is -Xmx 64m.

You should be able to reproduce by just starting with a base ApacheDS 
instance, adding a few thousand entries then searching for them.

I created an LDIF with 25000 users of the form:

dn: uid=user1,ou=users,ou=system
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: top
cn: user1
displayname: test user
sn: test
uid: test1

then I added them with: ldapmodify -h 127.0.0.1 -p 10389 -D 
uid=admin,ou=system -w secret -f users.ldif -a

I searched with: ldapsearch -h 127.0.0.1 -p 10389 -D uid=admin,ou=system 
-w secret -z 2000 -b ou=users,ou=system -s one -LLL (objectclass=*) 1.1

This has a delay of a few seconds before the 2000 results are shown. 
Searching without the size limit (-z 2000) gives the OutOfMemoryError 
without returning any results.

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Emmanuel Lecharny <el...@apache.org>.
Martin Alderson wrote:
> Hi Emmanuel,
>
>> There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the 
>> CircularList is not thread safe, and when writing messages into it, 
>> you may have impredictable results. I replaced it with a 
>> ConcurrentList, and the problem is now fixed. We still have to 
>> release MINA 2.0.0-RC1, but I think we will do maybe next week.
>>
>> Can you do that : build MINA trunk, and use it 
>> (MINA-2.0.0-M4-SNAPSHOT) for a new test, to see if the 
>> SearchResponseEntry are still hold ?
>>
>> Thanks !
>
> I've just built mina trunk (producing 
> mina-core-2.0.0-RC1-SNAPSHOT.jar) and tested with that.
>
> Unfortunately it still does the same, i.e. the SearchResponseEntrys 
> are held by the DefaultWriteRequestQueue.
Ok, so we still have a serious issu we have to fix quickly...

Can you give us the information about the used JVM, memory used and such 
so that we can reproduce the problem easilly ?

May be we should reopen the JIRA too.

Thanks !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <eq...@planetquake.com>.
Hi Emmanuel,

> There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the CircularList 
> is not thread safe, and when writing messages into it, you may have 
> impredictable results. I replaced it with a ConcurrentList, and the 
> problem is now fixed. We still have to release MINA 2.0.0-RC1, but I 
> think we will do maybe next week.
> 
> Can you do that : build MINA trunk, and use it (MINA-2.0.0-M4-SNAPSHOT) 
> for a new test, to see if the SearchResponseEntry are still hold ?
> 
> Thanks !

I've just built mina trunk (producing mina-core-2.0.0-RC1-SNAPSHOT.jar) 
and tested with that.

Unfortunately it still does the same, i.e. the SearchResponseEntrys are 
held by the DefaultWriteRequestQueue.

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Emmanuel Lecharny <el...@apache.org>.
Martin Alderson wrote:
>> There was a problem in MINA 1.1.7 in 1.5.4. In fact, if you do a big
>> search request, responses are stacked into MINA until the searchResultDone
>> is found. It has been fixed while moving to MINA 2.0.
>> So now, in trunk if you still experience a OOM, then we should investigate
>> what happens.
>>
>> We also fixed a memory leak somewhere else (don't remember, but Norval
>> Hope found it and we applied the patch)
>>
>> Can you do some profiling session with Yourkit to see where the memory is
>> being sucked ?
>>     
>
> I've done some profiling with JMP and found that 1.5.1, 1.5.4 and trunk all seem to be holding on to the SearchResponseEntryImpl objects until the end of the search.
>
>
> For ApacheDS 1.5.4 and 1.5.1 the SearchResponseEntryImpl objects owner graph looks like:
>
> org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
>   owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer.message
>     owned by org.apache.mina.filter.executor.ExecutorFilter$Event.data
>       owned by java.util.LinkedList$Entry.element
>         owned by java.util.LinkedList$Entry.(next, previous)
>
> In ApacheDS trunk (as of 31st March):
>
> org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
>   owned by org.apache.mina.core.write.DefaultWriteRequest.message
>     owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageWriteRequest.parentRequest
>       owned by Object[1490]
>         owned by org.apache.mina.util.CircularQueue.items
>           owned by org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue.q
>
>
> Strangely ApacheDS 1.5.4 actually streams the search responses to the client immediately - but still keeps the SearchResponseEntryImpl objects in memory.
>
> Any ideas are appreciated!
>   
There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the CircularList 
is not thread safe, and when writing messages into it, you may have 
impredictable results. I replaced it with a ConcurrentList, and the 
problem is now fixed. We still have to release MINA 2.0.0-RC1, but I 
think we will do maybe next week.

Can you do that : build MINA trunk, and use it (MINA-2.0.0-M4-SNAPSHOT) 
for a new test, to see if the SearchResponseEntry are still hold ?

Thanks !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

RE: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <Ma...@salfordsoftware.co.uk>.
> There was a problem in MINA 1.1.7 in 1.5.4. In fact, if you do a big
> search request, responses are stacked into MINA until the searchResultDone
> is found. It has been fixed while moving to MINA 2.0.
> So now, in trunk if you still experience a OOM, then we should investigate
> what happens.
> 
> We also fixed a memory leak somewhere else (don't remember, but Norval
> Hope found it and we applied the patch)
> 
> Can you do some profiling session with Yourkit to see where the memory is
> being sucked ?

I've done some profiling with JMP and found that 1.5.1, 1.5.4 and trunk all seem to be holding on to the SearchResponseEntryImpl objects until the end of the search.


For ApacheDS 1.5.4 and 1.5.1 the SearchResponseEntryImpl objects owner graph looks like:

org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
  owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer.message
    owned by org.apache.mina.filter.executor.ExecutorFilter$Event.data
      owned by java.util.LinkedList$Entry.element
        owned by java.util.LinkedList$Entry.(next, previous)

In ApacheDS trunk (as of 31st March):

org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
  owned by org.apache.mina.core.write.DefaultWriteRequest.message
    owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageWriteRequest.parentRequest
      owned by Object[1490]
        owned by org.apache.mina.util.CircularQueue.items
          owned by org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue.q


Strangely ApacheDS 1.5.4 actually streams the search responses to the client immediately - but still keeps the SearchResponseEntryImpl objects in memory.

Any ideas are appreciated!

Thanks,

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Emmanuel Lecharny <el...@apache.org>.
Martin Alderson wrote:
> I've tried trunk now and though it is slightly different I still get the same OutOfMemoryError.
>
> With 1.5.4 it was as if the server was sending the entries to the client but caching them somewhere - so the client would receive a few thousand results then the server would hit the OutOfMemoryError.  Now no entries are returned (unless a low size limit is specified), as if the server is building up the full list of entries in memory before attempting to send any of them to the client.
>
> Any thoughts?
>   
There was a problem in MINA 1.1.7 in 1.5.4. In fact, if you do a big 
search request, responses are stacked into MINA until the 
searchResultDone is found. It has been fixed while moving to MINA 2.0. 
So now, in trunk if you still experience a OOM, then we should 
investigate what happens.

We also fixed a memory leak somewhere else (don't remember, but Norval 
Hope found it and we applied the patch)

Can you do some profiling session with Yourkit to see where the memory 
is being sucked ?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

RE: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <Ma...@salfordsoftware.co.uk>.
I've tried trunk now and though it is slightly different I still get the same OutOfMemoryError.

With 1.5.4 it was as if the server was sending the entries to the client but caching them somewhere - so the client would receive a few thousand results then the server would hit the OutOfMemoryError.  Now no entries are returned (unless a low size limit is specified), as if the server is building up the full list of entries in memory before attempting to send any of them to the client.

Any thoughts?

Thanks,

Martin



> Ok working now.  Yeah I remember this.  I think this was before we
> switched over to MINA 2.0. But yeah we did not apply this and I don't know
> if it was.  Emmanuel would know best of all since he seemed to be keeping
> tabs on this.  I'll ask him about this issues shortly here.
> 
> Alex
> 
> On Wed, Mar 25, 2009 at 7:33 PM, Martin Alderson
> <eq...@planetquake.com>wrote:
> 
> > Hi Alex,
> >
> > I'm using the default which seems to be -Xmx64m.
> >
> > I've just remembered and dug up a JIRA from last year that looks
> > likely to be the cause of this:
> > https://issues.apache.org/jira/browse/DIRSERVER-1161
> >
> > This looks to be fixed but unfortunately not yet released.  I'll
> > either have to upgrade my projects version of ApacheDS to trunk which
> > I'm not too keen on (and would be a fair bit of work as I have
> > interceptors built for ApacheDS 1.5.1), or backport the fix from that
> JIRA.
> >
> > The fix is mentioned as "adding an ExecutorFilter on the WRITE
> EventType"
> > which was made to LdapService.  This class didn't exist in ApacheDS
> > 1.5.1 so does anyone know where the equivalent code was back then (no
> > worries if not, I should be able to trace the SVN logs back)? Also
> > does this fix depend upon MINA 2?
> >
> > Thanks,
> >
> > Martin
> >
> >
> >
> > Alex Karasulu wrote:
> >
> >> How much memory are you running with on your JVM?
> >>
> >> Alex
> >>
> >> On Wed, Mar 25, 2009 at 6:07 PM, Martin Alderson
> >> <equim@planetquake.com
> >> >wrote:
> >>
> >>  Hi all,
> >>>
> >>> I'm just running some tests on ApacheDS 1.5.4 on Windows.
> >>>
> >>> I have added 25000 simple users to ou=users,ou=system [1].  After
> >>> running the following LDAP search command I get between 1789 and
> >>> 5179 results each time before an OutOfMemoryError is produced by the
> >>> server.
> >>>
> >>> I'm not really sure why this is happening - shouldn't each search
> >>> result entry be discarded after it has been sent back to the client?
> >>>
> >>> Thanks,
> >>>
> >>> Martin
> >>>
> >>>
> >>> [1] The format of the users added is:
> >>> dn: uid=user1,ou=users,ou=system
> >>> objectclass: person
> >>> objectclass: organizationalPerson
> >>> objectclass: inetOrgPerson
> >>> objectclass: top
> >>> cn: user1
> >>> displayname: test user
> >>> sn: test
> >>> uid: test1
> >>>
> >>>
> >>
> >

Re: [ApacheDS] OutOfMemoryError

Posted by Alex Karasulu <ak...@gmail.com>.
Ok working now.  Yeah I remember this.  I think this was before we switched
over to MINA 2.0. But yeah we did not apply this and I don't know if it
was.  Emmanuel would know best of all since he seemed to be keeping tabs on
this.  I'll ask him about this issues shortly here.

Alex

On Wed, Mar 25, 2009 at 7:33 PM, Martin Alderson <eq...@planetquake.com>wrote:

> Hi Alex,
>
> I'm using the default which seems to be -Xmx64m.
>
> I've just remembered and dug up a JIRA from last year that looks likely to
> be the cause of this: https://issues.apache.org/jira/browse/DIRSERVER-1161
>
> This looks to be fixed but unfortunately not yet released.  I'll either
> have to upgrade my projects version of ApacheDS to trunk which I'm not too
> keen on (and would be a fair bit of work as I have interceptors built for
> ApacheDS 1.5.1), or backport the fix from that JIRA.
>
> The fix is mentioned as "adding an ExecutorFilter on the WRITE EventType"
> which was made to LdapService.  This class didn't exist in ApacheDS 1.5.1 so
> does anyone know where the equivalent code was back then (no worries if not,
> I should be able to trace the SVN logs back)? Also does this fix depend upon
> MINA 2?
>
> Thanks,
>
> Martin
>
>
>
> Alex Karasulu wrote:
>
>> How much memory are you running with on your JVM?
>>
>> Alex
>>
>> On Wed, Mar 25, 2009 at 6:07 PM, Martin Alderson <equim@planetquake.com
>> >wrote:
>>
>>  Hi all,
>>>
>>> I'm just running some tests on ApacheDS 1.5.4 on Windows.
>>>
>>> I have added 25000 simple users to ou=users,ou=system [1].  After running
>>> the following LDAP search command I get between 1789 and 5179 results
>>> each
>>> time before an OutOfMemoryError is produced by the server.
>>>
>>> I'm not really sure why this is happening - shouldn't each search result
>>> entry be discarded after it has been sent back to the client?
>>>
>>> Thanks,
>>>
>>> Martin
>>>
>>>
>>> [1] The format of the users added is:
>>> dn: uid=user1,ou=users,ou=system
>>> objectclass: person
>>> objectclass: organizationalPerson
>>> objectclass: inetOrgPerson
>>> objectclass: top
>>> cn: user1
>>> displayname: test user
>>> sn: test
>>> uid: test1
>>>
>>>
>>
>

Re: [ApacheDS] OutOfMemoryError

Posted by Alex Karasulu <ak...@gmail.com>.
Seems like the site is down now (JIRA).  Will check it out again later.

Alex

On Wed, Mar 25, 2009 at 7:33 PM, Martin Alderson <eq...@planetquake.com>wrote:

> Hi Alex,
>
> I'm using the default which seems to be -Xmx64m.
>
> I've just remembered and dug up a JIRA from last year that looks likely to
> be the cause of this: https://issues.apache.org/jira/browse/DIRSERVER-1161
>
> This looks to be fixed but unfortunately not yet released.  I'll either
> have to upgrade my projects version of ApacheDS to trunk which I'm not too
> keen on (and would be a fair bit of work as I have interceptors built for
> ApacheDS 1.5.1), or backport the fix from that JIRA.
>
> The fix is mentioned as "adding an ExecutorFilter on the WRITE EventType"
> which was made to LdapService.  This class didn't exist in ApacheDS 1.5.1 so
> does anyone know where the equivalent code was back then (no worries if not,
> I should be able to trace the SVN logs back)? Also does this fix depend upon
> MINA 2?
>
> Thanks,
>
> Martin
>
>
>
> Alex Karasulu wrote:
>
>> How much memory are you running with on your JVM?
>>
>> Alex
>>
>> On Wed, Mar 25, 2009 at 6:07 PM, Martin Alderson <equim@planetquake.com
>> >wrote:
>>
>>  Hi all,
>>>
>>> I'm just running some tests on ApacheDS 1.5.4 on Windows.
>>>
>>> I have added 25000 simple users to ou=users,ou=system [1].  After running
>>> the following LDAP search command I get between 1789 and 5179 results
>>> each
>>> time before an OutOfMemoryError is produced by the server.
>>>
>>> I'm not really sure why this is happening - shouldn't each search result
>>> entry be discarded after it has been sent back to the client?
>>>
>>> Thanks,
>>>
>>> Martin
>>>
>>>
>>> [1] The format of the users added is:
>>> dn: uid=user1,ou=users,ou=system
>>> objectclass: person
>>> objectclass: organizationalPerson
>>> objectclass: inetOrgPerson
>>> objectclass: top
>>> cn: user1
>>> displayname: test user
>>> sn: test
>>> uid: test1
>>>
>>>
>>
>

Re: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <eq...@planetquake.com>.
Hi Alex,

I'm using the default which seems to be -Xmx64m.

I've just remembered and dug up a JIRA from last year that looks likely 
to be the cause of this: 
https://issues.apache.org/jira/browse/DIRSERVER-1161

This looks to be fixed but unfortunately not yet released.  I'll either 
have to upgrade my projects version of ApacheDS to trunk which I'm not 
too keen on (and would be a fair bit of work as I have interceptors 
built for ApacheDS 1.5.1), or backport the fix from that JIRA.

The fix is mentioned as "adding an ExecutorFilter on the WRITE 
EventType" which was made to LdapService.  This class didn't exist in 
ApacheDS 1.5.1 so does anyone know where the equivalent code was back 
then (no worries if not, I should be able to trace the SVN logs back)? 
Also does this fix depend upon MINA 2?

Thanks,

Martin


Alex Karasulu wrote:
> How much memory are you running with on your JVM?
> 
> Alex
> 
> On Wed, Mar 25, 2009 at 6:07 PM, Martin Alderson <eq...@planetquake.com>wrote:
> 
>> Hi all,
>>
>> I'm just running some tests on ApacheDS 1.5.4 on Windows.
>>
>> I have added 25000 simple users to ou=users,ou=system [1].  After running
>> the following LDAP search command I get between 1789 and 5179 results each
>> time before an OutOfMemoryError is produced by the server.
>>
>> I'm not really sure why this is happening - shouldn't each search result
>> entry be discarded after it has been sent back to the client?
>>
>> Thanks,
>>
>> Martin
>>
>>
>> [1] The format of the users added is:
>> dn: uid=user1,ou=users,ou=system
>> objectclass: person
>> objectclass: organizationalPerson
>> objectclass: inetOrgPerson
>> objectclass: top
>> cn: user1
>> displayname: test user
>> sn: test
>> uid: test1
>>
>

Re: [ApacheDS] OutOfMemoryError

Posted by Alex Karasulu <ak...@gmail.com>.
How much memory are you running with on your JVM?

Alex

On Wed, Mar 25, 2009 at 6:07 PM, Martin Alderson <eq...@planetquake.com>wrote:

> Hi all,
>
> I'm just running some tests on ApacheDS 1.5.4 on Windows.
>
> I have added 25000 simple users to ou=users,ou=system [1].  After running
> the following LDAP search command I get between 1789 and 5179 results each
> time before an OutOfMemoryError is produced by the server.
>
> I'm not really sure why this is happening - shouldn't each search result
> entry be discarded after it has been sent back to the client?
>
> Thanks,
>
> Martin
>
>
> [1] The format of the users added is:
> dn: uid=user1,ou=users,ou=system
> objectclass: person
> objectclass: organizationalPerson
> objectclass: inetOrgPerson
> objectclass: top
> cn: user1
> displayname: test user
> sn: test
> uid: test1
>